DOWNSTREAM TASK

Revealing Weaknesses of Vietnamese Language Models Through Unanswerable Questions in Machine Reading Comprehension (2023.eacl-srw)

Challenge:	Automatic Speech Recognition (ASR) errors in voice-based dialog systems pose significant impediments to downstream tasks.
Approach:	They propose an automatic speech recognition (ASR) error-aware loss function to inject failed mentions and resolved entity names into the knowledge graph to enhance its awareness of unresolved mentions.
Outcome:	The proposed system enhances the knowledge graph's awareness of unresolved mentions by injecting pairs of failed mentions and resolved entities into the knowledge map.

Copied to clipboard

Son Quoc Tran, Phong Nguyen-Thuan Do, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen

DynaMaR: Dynamic Prompt with Mask Token Representation (2022.emnlp-industry)

Challenge:	Existing problems in Vietnamese Machine Reading Comprehension systems are limited due to multilinguality, which limits the ability of multilingual models to develop state-of-the-art systems.
Approach:	They propose to modify the process of annotating unanswerable questions to improve the quality of unanswered questions to a higher level of difficulty for Machine Reading Comprehension systems to solve.
Outcome:	The proposed modification improves the quality of unanswerable questions to a higher level of difficulty for Machine Reading Comprehension systems to solve.

Copied to clipboard

Xiaodi Sun, Sunny Rajagopalan, Priyanka Nigam, Weiyi Lu, Yi Xu, Iman Keivanloo, Belinda Zeng, Trishul Chilimbi

An Explainable Toolbox for Evaluating Pre-trained Vision-Language Models (2022.emnlp-demos)

Challenge:	Recent research shows that large language models pretrained using unsupervised approaches can achieve significant performance improvement on many downstream tasks.
Approach:	They propose an unsupervised approach to fine-tuning large language models using unsupervised approaches to many downstream tasks.
Outcome:	The proposed approach improves on four e-commerce applications and can achieve an average improvement of 10% in few-shot settings and 3.7% in data-rich settings over the standard approach.

Copied to clipboard

Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, Jianwei Yin

How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese (2023.acl-srw)

Challenge:	Existing studies evaluate VLP models by comparing the fine-tuned downstream task performance with the average downstream task accuracy.
Approach:	They propose a toolbox for evaluating Vision-Language Pretraining (VLP) models.
Outcome:	The proposed toolbox provides the preliminary datasets that deepen the image-texting ability of a VLP model.

Copied to clipboard

Takuro Fujii, Koki Shibata, Atsuki Yamaguchi, Terufumi Morishita, Yasuhiro Sogawa

HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization (2025.acl-srw)

Challenge:	Existing studies on scriptio continua languages lack comprehensiveness of tokenizers . authors use Byte-Pair-Encoding or Unigram instead of WordPiece for subword tokenizer .
Approach:	They investigate the effect of tokenizers on the downstream performance of pretrained language models in scriptio continua languages where no explicit spaces exist between words.
Outcome:	The proposed tokenizers perform better on a wide range of tasks compared with other tokenizer methods . the results show that each task has an optimal morphological analyzer .

Copied to clipboard

Enes Özeren, Yihong Liu, Hinrich Schuetze

Commonsense Reasoning for Natural Language Processing (2020.acl-tutorials)

Challenge:	Pre-trained language models exhibit suboptimal performance on mid- and low-resource languages due to limited exposure to these languages during pre-training.
Approach:	They propose a similarity-based subword embedding initialization heuristic that introduces new tokens specific to target languages, initializes their embedders, and applies continual pre-training on target-language data.
Outcome:	The proposed method outperforms random initialization baseline and matches or exceeds OFA in both continual pre-training convergence and downstream task performance.

Copied to clipboard

Maarten Sap, Vered Shwartz, Antoine Bosselut, Yejin Choi, Dan Roth

On the Limitations of Language-targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning (2026.tacl-1)

Challenge:	In this tutorial, we will outline the various types of commonsense knowledge and discuss techniques to gather and represent commonsence knowledge.
Approach:	This tutorial will provide researchers with the critical foundations and recent advances in commonsense representation and reasoning.
Outcome:	This tutorial will outline the various types of commonsense and discuss techniques to gather and represent commonsence knowledge while highlighting the challenges specific to this type of knowledge (e.g., reporting bias).

Copied to clipboard

Simon Kurz, Jian-Jia Chen, Lucie Flek, Zhixue Zhao

Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks (2022.emnlp-industry)

Challenge:	Recent advances in large language model pruning have shown high predictive performance in post-training settings.
Approach:	They conduct an empirical study on the performance and internal representation changes associated with pruning multilingual models for monolingual applications.
Outcome:	The proposed pruning methods retain perplexity and yield high signal-to-noise ratios, but not consistently improve downstream tasks.

Copied to clipboard

Charith Peris, Lizhen Tan, Thomas Gueudre, Turan Gojayev, Pan Wei, Gokmen Oz

Extreme Model Compression for On-device Natural Language Understanding (2020.coling-industry)

Challenge:	Domain Classification (DC) and Intent Classification/Named Entity Recognition (ICNER) are the most common methods for reducing teacher-student knowledge into manageable sizes for low-latency downstream applications.
Approach:	They investigate whether distillation from a generic LM benefits downstream tasks . a domain classification and a task-specific data set are used to fine tune the model .
Outcome:	The proposed model improves across tasks and test sets when only task-specific data is used.

Copied to clipboard

Kanthashree Mysore Sathyendra, Samridhi Choudhary, Leah Nicolich-Henkin

Challenge:	Xu and Sarikaya et al., 2014) perform word-embedding compression with NLU task learning . their approach achieves a compression rate of 97.4% with less than 3.7% degradation in predictive performance.
Approach:	They propose a task-aware, end-to-end compression approach that performs word-embedding compression with NLU task learning.
Outcome:	The proposed approach outperforms baselines and achieves 97.4% compression rate with less than 3.7% degradation in predictive performance.

Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages (2022.acl-long)

Copied to clipboard

Vaidehi Patil, Partha Talukdar, Sunita Sarawagi

Low-Resource NLP

Challenge:	Pre-trained multilingual models have shown great potential for zero-shot cross-lingual transfer to low web-resource languages (LRLs).
Approach:	They propose a vocabulary generation algorithm which enhances lexical overlap across related languages by generating a token that increases the representation of LRLs.
Outcome:	The proposed approach improves cross-lingual transfer accuracy without reducing HRL representation and accuracy.

MGPO: Thinking with Images via Multi-Turn Grounding-Based Reinforcement Learning (2026.findings-acl)

Copied to clipboard

Xinyu Huang, Yuhao Dong, Weiwei Tian, Bo Li, Rui Feng, Ziwei Liu

A Multi-Level Optimization Framework for End-to-End Text Augmentation (2022.tacl-1)

Challenge:	State-of-the-art large multimodal models face challenges when processing high-resolution images, as these inputs are converted into enormous visual tokens, many of which are irrelevant to the downstream task.
Approach:	They propose a multi-turn grounding-based policy optimization framework that enables LMMs to iteratively focus on key visual regions by automatically cropping sub-images based on model-predicted grounding coordinates within a multiple-turn conversation framework.
Outcome:	The proposed framework improves on Qwen2.5-VL-7B with 21K samples and surpasses OpenAI’s o1 and GPT-4o models on the out-of-distribution (OOD) V* Bench.

Copied to clipboard

Sai Ashish Somayajula, Linfeng Song, Pengtao Xie

Joint Optimization of Tokenization and Downstream Model (2021.findings-acl)

Challenge:	Existing methods for text augmentation perform data augmentation and downstream tasks separately.
Approach:	They propose a framework to perform text augmentation and the downstream task end-to-end.
Outcome:	The proposed framework performs text augmentation and the downstream task end-to-end on a text classification dataset.

Copied to clipboard

Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, Naoaki Okazaki

Text Segmentation

Challenge:	Existing studies have reported that an appropriate tokenization depends on each downstream task.
Approach:	They propose a method to find an appropriate tokenization to a downstream task by optimizing a tokenizer and a model.
Outcome:	The proposed method improves on text classification and machine translation tasks.

BMInf: An Efficient Toolkit for Big Model Inference and Tuning (2022.acl-demo)

Copied to clipboard

Xu Han, Guoyang Zeng, Weilin Zhao, Zhiyuan Liu, Zhengyan Zhang, Jie Zhou, Jun Zhang, Jia Chao, Maosong Sun

Subasa - Adapting Language Models for Low-resourced Offensive Language Detection in Sinhala (2025.naacl-srw)

Challenge:	Recent years, pre-trained language models (PLMs) have achieved promising results on various NLP tasks.
Approach:	They propose an open-source toolkit for big model inference and tuning which can support big model tuning at extremely low computation cost.
Outcome:	The proposed toolkit can support big model inference and tuning at extremely low computation cost.

Copied to clipboard

Shanilka Haturusinghe, Tharindu Cyril Weerasooriya, Christopher M Homan, Marcos Zampieri, Sidath Ravindra Liyanage

MVP-BERT: Multi-Vocab Pre-training for Chinese BERT (2021.acl-srw)

Challenge:	A major challenge in the field of NLP are the disparities between high- and low-resource languages.
Approach:	They propose fine-tuning strategies that have not been previously explored for Sinhala in the downstream task of offensive language detection.
Outcome:	The proposed models outperform baseline models on the Sinhala offensive language detection task.

Copied to clipboard

Wei Zhu

Prompt Tuning for Unified Multimodal Pretrained Models (2023.findings-acl)

Challenge:	Pre-trained language models (PLMs) improve performance of Chinese NLP tasks, but the vocabulary (vocab) for these Chinese PLMs remains to be the one provided by Google Chinese BERT (CITATION), which is based on Chinese characters (chars).
Approach:	They propose two versions of multi-vocab pre-training (MVP) to improve Chinese PLMs' expressiveness.
Outcome:	The proposed model outperforms the recent AMBERT (CITATION) model on span-level tasks and is more robust against adversarial attacks.

Copied to clipboard

Hao Yang, Junyang Lin, An Yang, Peng Wang, Chang Zhou

Bootstrapping Small & High Performance Language Models with Unmasking-Removal Training Policy (2023.emnlp-main)

Challenge:	Prompt tuning has demonstrated success in natural language pretraining and even vision pretraining.
Approach:	They propose to apply prompt tuning to a unified sequence-to-sequence pretrained model by adding a sequence of learnable embeddings to each layer and finetuning the pretrained models on downstream tasks.
Outcome:	The proposed method outperforms other parameter-efficient tuning methods on multimodal models and is robust against adversarial attacks.

Copied to clipboard

Yahan Yang, Elior Sulem, Insup Lee, Dan Roth

Iterative Paraphrastic Augmentation with Discriminative Span Alignment (2021.tacl-1)

Challenge:	Large-scale pre-trained language models (LMs) have shown promising ability on handling various downstream tasks including textual classification and question answering.
Approach:	They propose to use BabyBERTa to train child-directed speech without unmasking words while masking parameters to improve grammatical accuracy.
Outcome:	The proposed model achieves grammatical ability comparable to RoBERTa-base model, which is trained on 6,000 times more words and 15 times more parameters.

Copied to clipboard

Ryan Culkin, J. Edward Hu, Elias Stengel-Eskin, Guanghui Qin, Benjamin Van Durme

Paraphrasing

Challenge:	Existing datasets can be expanded or created using a small, manually produced seed corpus.
Approach:	They propose a paraphrastic augmentation strategy based on sentence-level lexically constrained paraphrases and discriminative span alignment.
Outcome:	The proposed approach allows for the large-scale expansion of existing datasets or the rapid creation of new datasets using a small, manually produced seed corpus.

HybridBERT - Making BERT Pretraining More Efficient Through Hybrid Mixture of Attention Mechanisms (2024.naacl-srw)

Copied to clipboard

Gokul Srinivasagan, Simon Ostermann

AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answering (2026.acl-long)

Challenge:	Pretrained transformer-based language models have produced state-of-the-art performance in most natural language understanding tasks.
Approach:	They propose two hybrid architectures that combine self-attention and additive attention mechanisms with sub-layer normalization to achieve double the pretraining accuracy of a vanilla-BERT baseline.
Outcome:	The proposed architectures outperform BERT-base on two downstream tasks while accelerating inference.

Copied to clipboard

Zheyuan Zhang, Kaiwen Shi, Zhengqing Yuan, Zehong Wang, Tianyi Ma, Keerthiram Murugesan, Vincent Galassi, Chuxu Zhang, Yanfang Ye

Unifying Parsing and Tree-Structured Models for Generating Sentence Semantic Representations (2022.naacl-srw)

Challenge:	Existing approaches to agent routing emphasize cost efficiency while overlooking the fine-grained contextual and relational structure inherent in QA tasks.
Approach:	They propose a framework that formulates multi-agent QA as a knowledge-graph-guided routing problem supervised by empirical performance signals.
Outcome:	The proposed framework outperforms single-agent and ensemble baselines while generalizing across benchmarks and LLM backbones.

Copied to clipboard

Antoine Simoulin, Benoit Crabbé

Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders (2024.acl-long)

Challenge:	Existing tree-based models require handannotated data to be trained.
Approach:	They propose a tree-based model that learns its composition function together with its structure.
Outcome:	The proposed model outperforms existing models on downstream tasks and is competitive with Bert base model.

Copied to clipboard

Yuwei Zhang, Siffi Singh, Sailik Sengupta, Igor Shalyminov, Hang Su, Hwanjun Song, Saab Mansour

Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021.tacl-1)

Challenge:	Conversational systems often rely on embedding models for intent classification and intent clustering tasks.
Approach:	They propose a toolkit that gives a more holistic view of intent embedding models by considering three tasks– (1) intent classification, (2) intent clustering, and (3) a novel triplet task.
Outcome:	The proposed model improves on the linguistic dimensions while affecting performance on downstream task metrics.

Copied to clipboard

Lisa Anne Hendricks, John Mellor, Rosalia Schneider, Jean-Baptiste Alayrac, Aida Nematzadeh

FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation (2022.emnlp-demos)

Challenge:	Recent studies suggest multimodal transformer models learn rich visual-linguistic representations.
Approach:	They focus on dataset noise and language similarity to their downstream task . they find that models with a multimodal attention mechanism outperform deeper models with modality-specific attention mechanisms.
Outcome:	The proposed models outperform models with a multimodal attention mechanism on downstream tasks.

Copied to clipboard

Tanya Goyal, Junyi Jessy Li, Greg Durrett

Text Generation

Challenge:	Existing tools to evaluate long text outputs are lacking in the field of NLP . human rating and error analysis remains a crucial component for any evaluation of long text generation.
Approach:	They propose a web-based toolkit to collect fine-grained error annotations for long texts . they use a taxonomy to identify errors and assign them to text spans .
Outcome:	The proposed tool can be used to evaluate the coherence of long generated summaries.

MAFIA: Multi-Adapter Fused Inclusive Language Models (2024.eacl-long)

Copied to clipboard

Prachi Jain, Ashutosh Sathe, Varun Gumma, Kabir Ahuja, Sunayana Sitaram

KIMERA: Injecting Domain Knowledge into Vacant Transformer Heads (2022.lrec-1)

Challenge:	Pretrained Language Models (PLMs) are widely used in NLP for various tasks.
Approach:	They propose to modularly debias a pre-trained language model across multiple bias dimensions using structured knowledge and a large generative model.
Outcome:	The proposed model is able to debias a pre-trained language model across multiple bias dimensions in a semi-automated way.

Copied to clipboard

Benjamin Winter, Alexei Figueroa Rosero, Alexander Löser, Felix Alexander Gers, Amy Siu

Towards Simple and Efficient Task-Adaptive Pre-training for Text Classification (2022.aacl-short)

Challenge:	Recent studies show that transformer models lack specific domain knowledge and are under-performing in broad domains like the medical domain.
Approach:	They propose a method for retraining and instilling attention heads with structured domain knowledge by masking redundant attention heads.
Outcome:	The proposed method improves on seven datasets in the medical domain in information retrieval and clinical outcome prediction settings.

Copied to clipboard

Arnav Ladkat, Aamir Miyajiwala, Samiksha Jagadale, Rekha A. Kulkarni, Raviraj Joshi

Llama meets EU: Investigating the European political spectrum through the lens of LLMs (2024.naacl-short)

Challenge:	Large-scale pre-trained language models are extensively trained on massive heterogeneous datasets, known as pre-training datasets.
Approach:	They propose to use Domain Adaptive Pre-training and Task-Adaptive pre-training as intermediate steps before the final finetuning task to cover the target domain vocabulary.
Outcome:	The proposed approach is computationally efficient, with 78% fewer parameters trained during TAPT.

Copied to clipboard

Ilias Chalkidis, Stephanie Brandl

Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting (2023.emnlp-main)

Challenge:	Large Language Models inherit clear political leanings that have been shown to influence downstream task performance.
Approach:	They adapt Llama Chat to a European political context and audit its political leanings based on the EUandI questionnaire to analyze its political knowledge and ability to reason in context.
Outcome:	The proposed model is adapted from speeches of individual euro-parties from debates in the European Parliament to analyze its political leanings.

Copied to clipboard

Xi Ye, Greg Durrett

Advancing Topic Segmentation and Outline Generation in Chinese Texts: The Paragraph-level Topic Representation, Corpus, and Benchmark (2024.lrec-main)

Challenge:	Recent work shows how to prompt large language models with explanations to obtain strong performance on textual reasoning tasks.
Approach:	They propose to optimize explanation-infused prompts in a blackbox fashion by using leave-one-out schemes and a two-stage framework.
Outcome:	The proposed method improves prompts over crowdworker annotations and naive search strategies.

Copied to clipboard

Feng Jiang, Weihao Liu, Xiaomin Chu, Peifeng Li, Qiaoming Zhu, Haizhou Li

Bootstrapping Transliteration with Constrained Discovery for Low-Resource Languages (D18-1)

Challenge:	Compared with sentence-level topic structure, paragraph-level topics can grasp and understand the context of a document from a higher level.
Approach:	They propose a hierarchical paragraph-level topic structure representation with three layers to guide corpus construction.
Outcome:	The proposed method achieves the largest Chinese paragraph-level topic structure corpus, achieving high quality.

Copied to clipboard

Shyam Upadhyay, Jordan Kodner, Dan Roth

Neurocache: Efficient Vector Retrieval for Long-range Language Modeling (2024.naacl-long)

Challenge:	Existing approaches to transliteration generation require a large number of training examples.
Approach:	They propose a bootstrapping algorithm that uses constrained discovery to improve generation . they show that the model can be used with as few as 500 training examples .
Outcome:	The proposed method improves on nine languages written in a unique script.

Copied to clipboard

Ali Safaya, Deniz Yuret

Does Transliteration Help Multilingual Language Modeling? (2023.findings-eacl)

Challenge:	Recent research shows that retrieval-augmented models with shorter contexts (4K tokens) can match the performance of models with longer contexts (16K/32K token)
Approach:	They introduce an approach to extend the effective context size of large language models by using an external vector cache to store past states.
Outcome:	The proposed method improves on models trained from scratch and pre-trained models.

Copied to clipboard

Ibraheem Muhammad Moosa, Mahmud Elahi Akhter, Ashfia Binte Habib

Adapter-TST: A Parameter Efficient Method for Multiple-Attribute Text Style Transfer (2023.findings-emnlp)

Challenge:	Script diversity presents a challenge to Multilingual Language Models by reducing lexical overlap . Script diversification can be used to improve performance of MLLMs by transliterating closely related languages to a common script.
Approach:	They empirically measure the effect of transliteration on MLLMs by focusing on Indic languages . they find that transliterations benefit low-resource languages without negatively affecting high-resourced ones .
Outcome:	The proposed transliteration-based model learns sentences that are more similar across languages.

Copied to clipboard

Zhiqiang Hu, Nancy Chen, Roy Lee

PRewrite: Prompt Rewriting with Reinforcement Learning (2024.acl-short)

Challenge:	Existing studies explore performing text style transfer on attributes like age, gender, formality, politeness, and formality.
Approach:	They propose a framework that freezes the pre-trained model’s original parameters and enables the development of a multiple-attribute text style transfer model.
Outcome:	The proposed model outperforms state-of-the-art models on sentiment transfer and multiple-attribute transfer tasks with significantly less computational resources.

Copied to clipboard

Weize Kong, Spurthi Hombaiah, Mingyang Zhang, Qiaozhu Mei, Michael Bendersky

Text Generation

Challenge:	Prompt engineering is done manually in a trial-and-error ad-hoc fashion, authors say .
Approach:	They propose a method to rewrite an under-optimized prompt to a more effective prompt.
Outcome:	The proposed method rewrites an under-optimized prompt to a more effective prompt.

KESA: A Knowledge Enhanced Approach To Sentiment Analysis (2022.aacl-main)

Copied to clipboard

Qinghua Zhao, Shuai Ma, Shuo Ren

A Generative Language Model for Few-shot Aspect-Based Sentiment Analysis (2022.findings-naacl)

Challenge:	Recent work on injecting sentiment knowledge into pre-trained language models, but it is difficult to integrate external knowledge into PLMs.
Approach:	They propose two sentiment-aware auxiliary tasks to integrate sentiment knowledge into the objective of the downstream task.
Outcome:	The proposed tasks outperform baselines and complement existing sentiment-enhanced models.

Copied to clipboard

Ehsan Hosseini-Asl, Wenhao Liu, Caiming Xiong

Cross-lingual Few-Shot Learning on Unseen Languages (2022.aacl-main)

Challenge:	Pre-trained language models are often used to achieve state-of-the-art results . eval paper shows that generative language model can handle joint and multi-task settings .
Approach:	They propose to reformulate extraction and prediction tasks into a sequence generation task . they propose a generative language model with unidirectional attention that learns to accomplish the tasks via language generation .
Outcome:	The proposed model outperforms the state-of-the-art in few-shot and full-shot settings.

Copied to clipboard

Genta Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, Daniel Preotiuc-Pietro

Low-Resource NLP

Challenge:	Large pre-trained language models have demonstrated the ability to obtain good performance on downstream tasks with limited examples in resource-rich languages.
Approach:	They propose to use a downstream sentiment analysis task to analyze the effectiveness of several few-shot learning strategies across 12 languages, including 8 unseen languages, to compare results.
Outcome:	The proposed model, XLM-R, gives the best performance on a task with few examples in resource-rich languages.

Domain-aware Self-supervised Pre-training for Label-Efficient Meme Analysis (2022.aacl-main)

Copied to clipboard

Shivam Sharma, Mohd Khizir Siddiqui, Md. Shad Akhtar, Tanmoy Chakraborty

STAMP: Selective Task-Aware Mechanism for Text Privacy (2026.eacl-long)

Challenge:	Existing self-supervised learning strategies focus on uni-modal applications . a recent study shows that multimodality is a major challenge for multi-modal systems .
Approach:	They propose two self-supervised pre-training methods that employ off-the-shelf multi-modal hate-speech data . they also incorporate multiple specialized pretext tasks to cater to complex multi-modity representation learning .
Outcome:	The proposed methods outperform the baseline self-supervised learning strategies on the Memotion challenge and the HarMeme task.

Copied to clipboard

Fengwei Tian, Payel Bhattacharjee, Heidi Hanson, Geoffrey D Rubin, Joseph Y. Lo, Ravi Tandon

Attention Fusion: a light yet efficient late fusion mechanism for task adaptation in NLU (2022.findings-naacl)

Challenge:	Experimental evaluations on SQuAD, Yelp, and AG News datasets demonstrate that STAMP achieves superior privacy–utility trade-offs across varying per-token privacy budgets.
Approach:	They propose a new framework for task-aware text privatization that selectively allocates privacy budgets across tokens by jointly considering (i) each token’s importance to the downstream task and (ii) its privacy sensitivity.
Outcome:	The proposed framework achieves superior privacy–utility trade-offs on SQuAD, Yelp, and AG News datasets.

Copied to clipboard

Jin Cao, Chandana Satya Prakash, Wael Hamza

Expert Calibration Lens for Pruning Mixture of Experts (2026.acl-demo)

Challenge:	a recent study has shown that fine-tuning pre-trained models is parameter-inefficient and expensive.
Approach:	They propose a task-attuned token module which integrates pre-trained network representations into a pre-trainer.
Outcome:	The proposed model trains only 0.0009% of the parameters and is efficient during computation and scalable during deployment.

Copied to clipboard

Luis Frentzen Salim, Chia-Chun Wu, Tran Van Nhiem, Lun-Wei Ku, Yung-Hui Li

Challenge:	Expert pruning is a practical deployment technique for Mixture-of-Experts models . but its success depends heavily on the calibration set used for pruning .
Approach:	They propose a calibration tool that compares expert activations across datasets to predict calibration perturbations without running expensive pruning procedures.
Outcome:	The proposed system compares expert activations across datasets to predict calibration perturbations without running expensive pruning procedures.

Text-to-Code Generation with Modality-relative Pre-training (2024.eacl-long)

Copied to clipboard

Fenia Christopoulou, Guchun Zhang, Gerasimos Lampouras

One Sentence, Two Embeddings: Contrastive Learning of Explicit and Implicit Semantic Representations (2026.findings-eacl)

Challenge:	Large pre-trained language models have been applied to programming language tasks with great success, often through further pre-training of a strictly-natural language model.
Approach:	They propose to map programming language modalities into the same embedding space by separating embeddable spaces between modality and modality-relative training objectives.
Outcome:	The proposed model can be adapted and represented differently depending on which modality they belong to and to the ultimate benefit of the downstream task.

Copied to clipboard

Kohei Oda, Po-Min Chuang, Kiyoaki Shirai, Natthawut Kertkeidkachorn

Learning Adaptive Axis Attentions in Fine-tuning: Beyond Fixed Sparse Attention Patterns (2022.findings-acl)

Challenge:	Existing sentence embedding methods lack the ability to capture the implicit semantics of sentences.
Approach:	They propose a sentence embedding method that assigns two embeddables to each sentence . one represents the explicit semantics and the other represents the implicit semantics . results show DualCSE can effectively encode both explicit and implicit meanings - they argue .
Outcome:	The proposed method can effectively encode both explicit and implicit meanings and improve the performance of the downstream task.

Copied to clipboard

Zihan Wang, Jiuxiang Gu, Jason Kuen, Handong Zhao, Vlad Morariu, Ruiyi Zhang, Ani Nenkova, Tong Sun, Jingbo Shang

Feature-Adaptive and Data-Scalable In-Context Learning (2024.acl-long)

Challenge:	Adaptive Axis Attention learns different attention patterns for each task and model layer . sparse attention patterns do not improve the run time of the models but they reduce model memory requirements .
Approach:	They propose a method that learns different attention patterns for each Transformer layer . they propose 'adaptive axis attention' method that identifies important tokens .
Outcome:	The proposed method does not require pre-training to accommodate sparse attention patterns.

Copied to clipboard

Jiahao Li, Quan Wang, Licheng Zhang, Guoqing Jin, Zhendong Mao

Unsupervised Keyphrase Extraction via Interpretable Neural Networks (2023.findings-eacl)

Challenge:	In-context learning (ICL) is a popular way to stimulate LLM capabilities for downstream tasks due to context length constraints.
Approach:	They propose a feature-adaptive and data-scalable in-context learning framework which leverages task-adaptives to promote inference on the downstream task.
Outcome:	The proposed framework outperforms state-of-the-art methods on 10 datasets under different data settings and LLM scale.

Copied to clipboard

Rishabh Joshi, Vidhisha Balachandran, Emily Saldanha, Maria Glenski, Svitlana Volkova, Yulia Tsvetkov

The Diminishing Returns of Masked Language Models to Science (2023.findings-acl)

Challenge:	Prior approaches for unsupervised keyphrase extraction relied on heuristic notions of phrase importance via embedding clustering or graph centrality.
Approach:	They propose an approach which defines keyphrases as document phrases that are salient for predicting the topic of the document.
Outcome:	The proposed method alleviates the need for ad-hoc heuristics and achieves state-of-the-art results in scientific publications and news articles.

Copied to clipboard

Zhi Hong, Aswathy Ajith, James Pauloski, Eamon Duede, Kyle Chard, Ian Foster

LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compression (2025.findings-naacl)

Challenge:	Existing studies have shown that masked language models can improve downstream tasks by pretraining larger models for longer on more data.
Approach:	They empirically evaluate the extent to which these results extend to tasks in science by using 14 domain-specific transformer-based masked language models.
Outcome:	The proposed model can improve on 12 scientific tasks, but not all.

Copied to clipboard

Souvik Kundu, Anahita Bhiwandiwalla, Sungduk Yu, Phillip Howard, Tiep Le, Sharath Nittur Sridhar, David Cobbley, Hao Kang, Vasudev Lal

Robust Multilingual Part-of-Speech Tagging via Adversarial Training (N18-1)

Challenge:	LVLMs have been shown to perform well on simple uni-modal benchmarks, but their detailed study on multi-modal models is still lacking.
Approach:	They propose a framework to analyze the impact of compression on LVLMs on multi-modal input driven tasks.
Outcome:	The proposed framework analyzes the impact of compression on generative performance of large vision language models on multi-modal input driven tasks.

Copied to clipboard

Michihiro Yasunaga, Jungo Kasai, Dragomir Radev

How much do contextualized representations encode long-range context? (2025.findings-naacl)

Challenge:	Adversarial training (AT) is a powerful regularization method for neural networks, aiming to achieve robustness to input perturbations.
Approach:	They propose and analyze a neural POS tagging model that exploits adversarial training by training on unmodified and adversarials.
Outcome:	The proposed model improves overall tagging accuracy and prevents over-fitting in low resource languages and boosts tabbing accuracy for rare / unseen words.

Copied to clipboard

Simeng Sun, Cheng-Ping Hsieh

Language Models of Code are Few-Shot Commonsense Learners (2022.emnlp-main)

Challenge:	Existing studies of contextualized representations focus on short sequences of tens to hundreds of tokens, whereas modern language models handle hundreds of thousands of token in a single context window.
Approach:	They use a perturbation setup and a metric to capture contextualization of long-range patterns from the perspective of representation geometry.
Outcome:	The proposed model can encode long-range contexts, but it's not fully recurrent, the authors say . their results suggest improvements in existing language models .

Copied to clipboard

Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, Graham Neubig

On the Importance of Effectively Adapting Pretrained Language Models for Active Learning (2022.acl-short)

Challenge:	Existing approaches to generate graphs using pre-trained language models hinder their ability to generate them correctly.
Approach:	They propose to frame structured commonsense reasoning tasks as code generation tasks instead of serializing the output graph as a flat list of nodes and edges.
Outcome:	The proposed approach outperforms natural-language LMs in three natural language tasks even when the downstream task does not involve source code at all.

Copied to clipboard

Katerina Margatina, Loic Barrault, Nikolaos Aletras

BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla (2022.findings-naacl)

Challenge:	Recent active learning approaches in NLP use off-the-shelf pretrained language models (LMs) . a poor training strategy can be catastrophic for AL, authors argue .
Approach:	They propose to first adapt the pretrained LM to the target task and then use it for AL.
Outcome:	The proposed approach provides substantial data efficiency improvements compared to the standard fine-tuning approach.

Copied to clipboard

Abhik Bhattacharjee, Tahmid Hasan, Wasi Ahmad, Kazi Samin Mubasshir, Md Saiful Islam, Anindya Iqbal, M. Sohel Rahman, Rifat Shahriyar

Detecting Extraneous Content in Podcasts (2021.eacl-main)

Challenge:	Bangla is a widely spoken yet low-resource language in the NLP literature.
Approach:	They propose a BERT-based natural language understanding model pretrainable in Bangla, a widely spoken yet low-resource language in the NLP literature.
Outcome:	The proposed model outperforms multilingual and monolingual models on four NLU tasks covering text classification, sequence labeling, and span prediction.

Copied to clipboard

Sravana Reddy, Yongze Yu, Aasish Pappu, Aswin Sivaraman, Rezvaneh Rezapour, Rosie Jones

Speech & Audio in NLP

Challenge:	Podcast episodes often contain extraneous material interleaved within the audio and the written descriptions . authors present classifiers that leverage both textual and listening patterns to detect such content .
Approach:	They propose a classifier that leverages both textual and listening patterns to detect extraneous material in podcast descriptions and audio transcripts.
Outcome:	The proposed classifiers improve ROUGE scores and reduce extraneous content in podcast summarization tasks.

AoE: Angle-optimized Embeddings for Semantic Textual Similarity (2024.acl-long)

Copied to clipboard

Xianming Li, Jing Li

Learning Disentangled Representations for Natural Language Definitions (2023.findings-eacl)

Challenge:	Existing text embedding models neglect cosine saturation zones rendering vanishing gradients and hindering learning subtle semantic differences in text embeds.
Approach:	They propose a novel angle-optimized Embedding model that optimizes angle differences in complex space to explore similarity in saturation zones better.
Outcome:	The proposed model outperforms existing models neglecting cosine saturation zones on STS and MTEB benchmarks and can produce high-quality embeddings and benefit downstream tasks.

Copied to clipboard

Danilo Silva De Carvalho, Giangiacomo Mercatali, Yingji Zhang, André Freitas

Learning and Evaluating a Differentially Private Pre-trained Language Model (2021.findings-emnlp)

Challenge:	Disentangling the encodings of neural models is a fundamental aspect for improving interpretability, semantic control and downstream task performance in Natural Language Processing.
Approach:	They propose to use syntactic and semantic regularities in textual data to provide models with both structural biases and generative factors.
Outcome:	The proposed model outperforms baselines on several qualitative and quantitative benchmarks and improves the results in the downstream task of definition modeling.

Copied to clipboard

Shlomo Hoory, Amir Feder, Avichai Tendler, Sofia Erell, Alon Peled-Cohen, Itay Laish, Hootan Nakhost, Uri Stemmer, Ayelet Benjamini, Avinatan Hassidim, Yossi Matias

Robust Transfer Learning with Pretrained Language Models through Adapters (2021.acl-short)

Challenge:	Contextual language models have improved performance but can lead to information leakage .
Approach:	They propose a differentially-private word-piece algorithm that allows training a tailored domain-specific vocabulary while maintaining privacy.
Outcome:	The proposed model can guarantee privacy while maintaining good model performance.

Copied to clipboard

Wenjuan Han, Bo Pang, Ying Nian Wu

Graph Neural Networks for Multiparallel Word Alignment (2022.findings-acl)

Challenge:	Existing approaches to transfer learning with pretrained transformer-based language models are not robust and can be adversarial.
Approach:	They propose a simple yet effective adapter-based approach to fine-tune language models on downstream tasks.
Outcome:	The proposed approach improves stability and adversarial robustness in transfer learning to various downstream tasks.

Copied to clipboard

Ayyoob Imani, Lütfi Kerem Senel, Masoud Jalili Sabet, François Yvon, Hinrich Schuetze

Structured Data in NLP

Challenge:	Generally, word alignment algorithms only use bitext and do not make use of the fact that many parallel corpora are multiparallel.
Approach:	They propose a multiparallel word alignment graph and graph neural networks to exploit it . they add and remove edges from the initial alignments and generalize the model .
Outcome:	The proposed method outperforms previous work on three word alignment datasets and on a downstream task.

Harnessing Cross-lingual Features to Improve Cognate Detection for Low-resource Languages (2020.coling-main)

Copied to clipboard

Diptesh Kanojia, Raj Dabre, Shubham Dewangan, Pushpak Bhattacharyya, Gholamreza Haffari, Malhar Kulkarni

Optimizing Word Segmentation for Downstream Task (2020.findings-emnlp)

Challenge:	a study of 14 Indian languages shows that cognates can be detected by word embeddings . cognates are variants of the same lexical form across languages .
Approach:	They propose to use cross-lingual word embeddings to detect cognates among 14 Indian languages . they then evaluate the impact of their method on neural machine translation .
Outcome:	The proposed method improves on a dataset of 12 Indian languages . it also improves quality of the extracted cognates by up to 2.76 BLEU .

Copied to clipboard

Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, Naoaki Okazaki

In-Contextual Gender Bias Suppression for Large Language Models (2024.findings-eacl)

Challenge:	Existing methods to optimize tokenizations for downstream tasks are not suitable for traditional NLP.
Approach:	They propose a method to explore a tokenization appropriate for a downstream task . they train a model to assign a high probability to such appropriate tokenization based on the downstream task loss .
Outcome:	The proposed method improves sentiment analysis and textual entailment tasks . it is also integrated into state-of-the-art contextualized embeddings and reports a positive effect .

Copied to clipboard

Daisuke Oba, Masahiro Kaneko, Danushka Bollegala

Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training (2021.emnlp-main)

Challenge:	Prior work has proposed debiasing methods that require human labelled examples, data augmentation and fine-tuning of LLMs, which are computationally expensive.
Approach:	They propose to suppress gender biases by providing textual preambles from manually designed templates and real-world statistics without accessing model parameters.
Outcome:	The proposed methods suppress gender biases in English LLMs using a CrowsPairs dataset without accessing model parameters.

Copied to clipboard

Kuan-Hao Huang, Wasi Ahmad, Nanyun Peng, Kai-Wei Chang

Article Classification with Graph Neural Networks and Multigraphs (2024.lrec-main)

Challenge:	Pre-trained multilingual language encoders do not precisely align words and phrases across languages.
Approach:	They propose a learning strategy for training robust models by drawing connections between adversarial examples and failure cases of zero-shot cross-lingual transfer.
Outcome:	The proposed model can achieve good performance even if representations of different languages are not aligned well.

Copied to clipboard

Khang Ly, Yury Kashnitsky, Savvas Chamezopoulos, Valeria Krzhizhanovskaya

Text Classification

Challenge:	Existing and newly published articles require complex and complex pipelines to classify them into context-specific label taxonomies.
Approach:	They propose to enrich Graph Neural Network pipelines with multi-graph representations that encode multiple signals of article relatedness as distinct edge types.
Outcome:	The proposed methods improve the performance of a variety of GNN models compared to default graphs.

Enhancing Natural Language Representation with Large-Scale Out-of-Domain Commonsense (2022.findings-acl)

Copied to clipboard

Wanyun Cui, Xingran Chen

Adversarial Robustness of Prompt-based Few-Shot Learning for Natural Language Understanding (2023.findings-acl)

Challenge:	Using commonsense in text understanding tasks can cause catastrophic forgetting due to domain discrepancy . previous methods of using textual descriptions as extra input information cannot apply to large-scale commonsensing.
Approach:	They propose to use out-of-domain commonsense to enhance text representation . they propose to integrate commonsensense descriptions into large-scale models .
Outcome:	The proposed model can integrate commonsense descriptions and enhance them to the target text representation without pre-training on large-scale unsupervised corpora.

Copied to clipboard

Venkata Prabhakara Sarath Nookala, Gaurav Verma, Subhabrata Mukherjee, Srijan Kumar

Crawling The Internal Knowledge-Base of Language Models (2023.findings-eacl)

Challenge:	Recent few-shot learning methods focus on improving downstream task performance, but there is limited understanding of the adversarial robustness of such methods.
Approach:	They evaluate prompt-based FSL methods against fully fine-tuned models to better understand the impact of various factors towards robustness.
Outcome:	The proposed methods show that they are less robust in the face of adversarial perturbations than fully fine-tuned models.

Copied to clipboard

Roi Cohen, Mor Geva, Jonathan Berant, Amir Globerson

Machine Translation for Machines: the Sentiment Classification Use Case (D19-1)

Challenge:	Existing methods for representing factual knowledge in a language model are insufficient.
Approach:	They propose a procedure for “crawling” the internal knowledge-base of a language model by expanding a knowledge-graph around it.
Outcome:	The proposed method yields high precision graphs (82-92%) while emitting a reasonable number of facts per entity.

Copied to clipboard

Amirhossein Tebbifakhr, Luisa Bentivogli, Matteo Negri, Marco Turchi

Information Retrieval

Challenge:	Traditionally, machine translation (MT) pursues a "human-oriented" objective: generating fluent output for a downstream task.
Approach:	They propose a neural machine translation approach that uses weak feedback to generate translations that are best suited for a downstream task.
Outcome:	The proposed approach outperforms general-purpose models and reinforcement learning methods on German and Italian tweets.

Does the Correctness of Factual Knowledge Matter for Factual Knowledge-Enhanced Pre-trained Language Models? (2023.emnlp-main)

Copied to clipboard

Boxi Cao, Qiaoyu Tang, Hongyu Lin, Xianpei Han, Le Sun

Gradient-based Constrained Sampling from Language Models (2022.emnlp-main)

Challenge:	Existing work neither proves that pre-trained models successfully learn the injected factual knowledge nor proves there is a causal relation between injected knowledge and downstream performance improvements.
Approach:	They propose a counterfactual-based analysis framework to explore the causal effects of factual knowledge injection on the performance of language models within pretrain-finetune paradigm.
Outcome:	The proposed framework shows that factual knowledge injection is successful but correctness of injected knowledge only has limited effect on the models’ downstream performance.

Copied to clipboard

Sachin Kumar, Biswajit Paria, Yulia Tsvetkov

RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models (2021.acl-long)

Challenge:	Large pretrained language models are successful at generating fluent text but are notoriously hard to controllably sample from.
Approach:	They propose a sampling procedure that combines the log-likelihood of the language model with arbitrary constraints in a single energy function and then generates samples in . non-autoregressive manner.
Outcome:	The proposed method improves on text generation with soft and hard constraints and keyword-guided generation.

Copied to clipboard

Soumya Barikeri, Anne Lauscher, Ivan Vulić, Goran Glavaš

Vec2Sent: Probing Sentence Embeddings with Natural Language Generation (2020.coling-main)

Challenge:	Recent work has focused on measuring and mitigating bias in pretrained language models.
Approach:	They propose a dataset that measures and mitigates bias across gender,race, religion, and queerness . they compare REDDITBIAS to a widely used conversational DialoGPT model .
Outcome:	The proposed framework measures and mitigates bias across gender,race, religion, and queerness dimensions.

Copied to clipboard

Martin Kerscher, Steffen Eger

A Two-Stage Framework with Self-Supervised Distillation for Cross-Domain Text Classification (2024.lrec-main)

Challenge:	a new unsupervised probing task is able to retrieve black-box sentence embeddings . a variety of problems surround probing tasks, including manual construing .
Approach:	They propose a method to generate black-box sentence embeddings by conditionally generating from them . they also illustrate how the language generated from different encoders differs .
Outcome:	The proposed probing task improves the performance of black-box sentence embeddings . the proposed task is based on a conditional natural language generation approach .

Copied to clipboard

Yunlong Feng, Bohan Li, Libo Qin, Xiao Xu, Wanxiang Che

CL-ReLKT: Cross-lingual Language Knowledge Transfer for Multilingual Retrieval Question Answering (2022.findings-naacl)

Challenge:	Existing work on cross-domain text classification relies on domain-invariant features or task-agnostic features.
Approach:	They propose a two-stage framework for cross-domain text classification that leverages or reuses rich labeled data from the source domain and unlabeled data in the target domain.
Outcome:	The proposed framework achieves state-of-the-art on a public cross-domain text classification benchmark.

Copied to clipboard

Peerat Limkonchotiwat, Wuttikorn Ponwitayarat, Can Udomcharoenchaikit, Ekapol Chuangsuwanich, Sarana Nutanong

Natural Language Interfaces

Challenge:	Existing approaches to cross-lingual question answering use sentence embedding to map documents and questions in multiple languages . a novel cross-linguistic approach to cross language-retrieval question answering is proposed . our method outperforms competitors in 19 out of 21 settings of CL-ReQA .
Approach:	They propose a cross-lingual language knowledge transfer framework for cross-linguistic question answering . they use a multilingual sentence embedding technique to create a linguistic embeddable space .
Outcome:	The proposed method outperforms current state-of-the-art methods in 19 out of 21 settings of CL-ReQA.

On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation (2021.acl-long)

Copied to clipboard

Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, Luo Si

Backpropagating through Structured Argmax using a SPIGOT (P18-1)

Challenge:	Existing studies have shown that adapter-based tuning is more parameter-efficient than fine-tuning.
Approach:	They propose to add adapter modules to a pretrained language model and update the parameters of adapter module when learning on a downstream task.
Outcome:	The proposed method outperforms fine-tuning on low-resource and cross-lingual tasks and settings.

Copied to clipboard

Hao Peng, Sam Thomson, Noah A. Smith

Evaluating Word Embeddings with Categorical Modularity (2021.findings-acl)

Challenge:	Structured projection of intermediate gradients (SPIGOT) is a new method for backpropagating through neural networks . structure-based learning methods for natural language processing are increasingly dominated by end-to-end differentiable functions .
Approach:	They propose a structured projection of intermediate gradients method for backpropagating through neural networks that includes hard-decision structured predictions in intermediate layers.
Outcome:	The proposed method improves on two structured NLP pipelines: syntactic-then-semantic dependency parsing and semantic parser followed by sentiment classification.

Copied to clipboard

Sílvia Casacuberta, Karina Halevy, Damián Blasi

Unsupervised Dialog Structure Learning (N19-1)

Challenge:	Existing word embeddings use different bilingual supervision signals with varying levels of strength.
Approach:	They propose a graph modularity metric to measure word embedding quality . they use a set of 500 words belonging to 59 neurobiologically motivated semantic categories .
Outcome:	The proposed metric measures word embedding quality on monolingual and cross-lingual tasks.

Copied to clipboard

Weiyan Shi, Tiancheng Zhao, Zhou Yu

BiKT: Enabling Bidirectional Knowledge Transfer Between Pretrained Models and Sequential Downstream Tasks (2024.findings-emnlp)

Challenge:	Current dialog systems require human experts to design the dialog structure, which is time consuming and sometimes insufficient to satisfy various customer needs.
Approach:	They propose to extract dialog structure using a modified VRNN model with discrete latent vectors.
Outcome:	The proposed model outperforms existing models on the ability to predict unseen data and is faster and more effective in a reinforcement learning setting.

Copied to clipboard

Hang Zeng, Chaoyue Niu, Fan Wu, Shaojie Tang, Leihao Pei, Chengfei Lv, Guihai Chen

KAD: A Framework for Proxy-based Test-time Alignment with Knapsack Approximation Deferral (2026.eacl-long)

Challenge:	Existing frameworks adapt from initial pretrained model to each downstream task directly, but ignore sequential nature of downstream tasks and feedback effect on pretrained models.
Approach:	They propose a framework to enable bidirectional knowledge transfer between pretrained models and downstream tasks in rounds.
Outcome:	The proposed framework improves on 9 GLUE datasets and 6 SuperGLUEs.

Copied to clipboard

Ayoub Hammal, Pierre Zweigenbaum, Caio Corro

Challenge:	Existing studies have shown that large language models require further alignment to adhere to downstream task requirements and stylistic preferences.
Approach:	They propose a proxy-based test-time alignment method to circumvent alignment costs by reducing the token-specific deferral rule to 0-1 knapsack problem.
Outcome:	The proposed method improves both task performance and speculative decoding speed.

Parsivar: A Language Processing Toolkit for Persian (L18-1)

Copied to clipboard

Salar Mohtaj, Behnam Roshanfekr, Atefeh Zafarian, Habibollah Asghari

Challenge:	a preprocessing step is required to convert text into a standard format for NLP tasks.
Approach:	They propose a Persian preprocessing toolkit that performs various kinds of activities . they use a plagiarism detection system to exploit the proposed toolkit .
Outcome:	The proposed tool outperforms available Persian preprocessing tools by about 8 percent in terms of F1 . the proposed toolkit performs normalization, space correction, tokenization, stemming, parts of speech tagging and shallow parsing tasks.

CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models (2023.emnlp-main)

Copied to clipboard

Aitor Ormazabal, Mikel Artetxe, Eneko Agirre

Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding (2025.naacl-long)

Challenge:	Methods for adapting language models to new tasks and domains have traditionally assumed white-box access to the model and work by modifying its parameters.
Approach:	They propose a method for adapting large language models to new domains and tasks . they fine-tune a small white-box LM and combine it with a large black-box model at the probability level through a network, learned on a smaller validation set.
Outcome:	The proposed method improves performance in all cases, while using a domain expert 23x smaller.

Copied to clipboard

Junyi Ye, Ankan Dash, Wenpeng Yin, Guiling Wang

Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning (2022.naacl-main)

Challenge:	Flowcharts are typically presented as images, driving the trend of using vision-language models for end-to-end flowchart understanding.
Approach:	They propose a vision-language model (VLM) that generates textual representations from flowchart images and a textual Reasoner that performs question-answering based on the text representations.
Outcome:	Experiments on the FlowVQA and FlowLearn benchmarks demonstrate TextFlow’s state-of-the-art performance as well as its robustness.

Copied to clipboard

Vishakh Padmakumar, Leonard Lausen, Miguel Ballesteros, Sheng Zha, He He, George Karypis

Speaker Turn Modeling for Dialogue Act Classification (2021.findings-emnlp)

Challenge:	Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks.
Approach:	They aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning by increasing the number of tasks and incorporating smaller sets of related tasks.
Outcome:	The proposed model improves on unseen target tasks by increasing the scale of multi-task learning to incorporate more tasks and developing similarity metrics to incorporate tasks related to the target task.

Copied to clipboard

Zihao He, Leili Tavabi, Kristina Lerman, Mohammad Soleymani

End-to-End Speech Recognition and Disfluency Removal (2020.findings-emnlp)

Challenge:	Existing approaches to DA classification model utterances without incorporating the turn changes among speakers throughout the dialogue, thus treating it no different than non-interactive written text.
Approach:	They propose to integrate the turn changes in conversations among speakers when modeling DAs by learning conversation-invariant speaker turn embeddings to represent speaker turns in a conversation.
Outcome:	The proposed model captures semantics from the dialogue content while accounting for different speaker turns in a conversation.

Copied to clipboard

Paria Jamshid Lou, Mark Johnson

How Gender Debiasing Affects Internal Model Representations, and Why It Matters (2022.naacl-main)

Challenge:	Disfluency detection is usually an intermediate step between an automatic speech recognition system and a downstream task.
Approach:	They propose to train models to directly map disfluent speech into fluent transcripts without relying on a separate disfluency detection model.
Outcome:	The proposed models learn to generate fluent transcripts, but their performance is slightly worse than a baseline pipeline approach consisting of an ASR system and a specialized disfluency detection model.

Copied to clipboard

Hadas Orgad, Seraphina Goldfarb-Tarrant, Yonatan Belinkov

Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformers (2023.findings-acl)

Challenge:	Existing studies of gender bias in NLP focus on extrinsic or intrinsic bias, but the relationship between extrindic and intrinsic bias is relatively unknown.
Approach:	They propose a framework to measure extrinsic and intrinsic bias together and propose metric to measure debiasing and intrinsic debiases.
Outcome:	The proposed framework provides a comprehensive perspective on bias in NLP models, which can be applied to deploy NLP systems in a more informed manner.

Copied to clipboard

Felix Gaschi, Patricio Cerda, Parisa Rastin, Yannick Toussaint

A Structured Span Selector (2022.naacl-main)

Challenge:	despite lack of explicit cross-lingual training data, multilingual models can achieve cross-linguistic transfer.
Approach:	They find alignment is significantly correlated with cross-lingual transfer . they advocate for further research on realignment methods for smaller models .
Outcome:	The proposed method outperforms XLM-R Large in POS-tagging between English and Arabic by +15.8 accuracy.

Copied to clipboard

Tianyu Liu, Yuchen Jiang, Ryan Cotterell, Mrinmaya Sachan

Curricular Next Conversation Prediction Pretraining for Transcript Segmentation (2023.findings-eacl)

Challenge:	a typical approach to natural language processing tasks involves selecting text spans and making decisions about them.
Approach:	They propose a grammar-based structured span selection model which learns to make use of partial span annotations.
Outcome:	The proposed model improves on two popular span prediction tasks.

Copied to clipboard

Anvesh Rao Vijjini, Hanieh Deilamsalehy, Franck Dernoncourt, Snigdha Chaturvedi

SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains (2023.eacl-main)

Challenge:	Prior research on document segmentation has focused on segmenting documents such as Wikipedia articles.
Approach:	They propose to pretrain a model to identify consecutive conversations to address these challenges . they introduce a curriculum to Advanced NCP to make the task more relevant to the downstream task .
Outcome:	The proposed model outperforms previous models in speech recognition errors and is robust to speech recognition.

Copied to clipboard

Koustava Goswami, Lukas Lange, Jun Araki, Heike Adel

Confronting LLMs with Traditional ML: Rethinking the Fairness of Large Language Models in Tabular Classifications (2024.naacl-long)

Challenge:	Recent work shows promising results when prompting pre-trained language models, but in low-resource domains, the domain gap between the pre-training data and the downstream task is too large.
Approach:	They propose a method for prompting pre-trained language models using domain-specific keywords with a trainable gated prompt.
Outcome:	The proposed prompting method outperforms state-of-the-art prompting methods on three text classification benchmarks and shows that it reduces the need for domain-specific language model pre-training.

Copied to clipboard

Yanchen Liu, Srishti Gautam, Jiaqi Ma, Himabindu Lakkaraju

Text Classification

Challenge:	Recent studies suggest using large language models to make tabular classifications . however, LLMs have been shown to exhibit harmful social biases based on stereotypes and inequalities present in society.
Approach:	They propose to use large language models to make tabular classifications . they show that LLMs inherit biases from their training data .
Outcome:	The proposed models exhibit harmful biases that reflect stereotypes and inequalities in society.

To Pretrain or Not to Pretrain: Examining the Benefits of Pretrainng on Resource Rich Tasks (2020.acl-main)

Copied to clipboard

Sinong Wang, Madian Khabsa, Hao Ma

Cross-Cultural Similarity Features for Cross-Lingual Transfer Learning of Pragmatically Motivated Tasks (2021.eacl-main)

Challenge:	Existing studies on pretraining NLP models with variants of Masked Language Model (MLM) objectives have shown that the number of training samples used in the downstream task is limited.
Approach:	They propose to use MLM objectives to pretrain NLP models with variants of Masked Language Model (MLM) objectives to improve accuracy on downstream tasks.
Outcome:	The proposed model can reach a diminishing return point as the supervised data size increases significantly.

Copied to clipboard

Jimin Sun, Hwijeen Ahn, Chan Young Park, Yulia Tsvetkov, David R. Mortensen

Automating Legal Interpretation with LLMs: Retrieval, Generation, and Evaluation (2025.acl-long)

Challenge:	a large amount of work on cross-lingual transfer learning focused on typological and genealogical similarities between languages.
Approach:	They propose three features that capture cross-cultural similarities that manifest in linguistic patterns and quantify distinct aspects of language pragmatics.
Outcome:	The proposed features capture cross-cultural similarities manifest in linguistic patterns and quantify aspects of language pragmatics.

Copied to clipboard

Kangcheng Luo, Quzhe Huang, Cong Jiang, Yansong Feng

Explainability & Interpretability in NLP

Challenge:	a novel framework for automated legal interpretation is proposed to alleviate the burden on legal experts.
Approach:	They propose a framework for automated legal interpretation that uses large language models to extract concept-related information and interpret legal concepts.
Outcome:	The proposed framework eliminates the need for legal experts to interpret legal concepts . it uses large language models to extract concept-related information and interpret legal concept interpretations .

On Surgical Fine-tuning for Language Encoders (2023.findings-emnlp)

Copied to clipboard

Abhilasha Lodha, Gayatri Belapurkar, Saloni Chalkapurkar, Yuanming Tao, Reshmi Ghosh, Samyadeep Basu, Dmitrii Petrov, Soundararajan Srinivasan

Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media (2021.findings-emnlp)

Challenge:	preserving knowledge of target distribution by fine-tuning all layers can be expensive and may increase data volume requirements.
Approach:	They propose an efficient metric based on the diagonal of the Fisher information matrix (FIM score) to select the candidate layers for selective fine-tuning.
Outcome:	The proposed metric can select layers leading to strong performance on GLUE and SuperGLUE tasks and across distinct language encoders.

Copied to clipboard

Paul Röttger, Janet Pierrehumbert

AFT-Tab: Adversarial Fine-Tuning for Tabular Data Synthesis with Long Text Columns (2026.acl-long)

Challenge:	Language use differs between domains and even within a domain, language use changes over time.
Approach:	They propose to use social media comments to study temporal adaptations in pre-trained language models.
Outcome:	The proposed model performs better on past than on future test sets, whereas adapting to domain does not improve performance on the downstream task.

Copied to clipboard

Yuhao Zhang, Liang Yan, Shaoming Duan, Xinyu Zha, Jinhang Su, Peiyi Han, Chuanyi Liu

Demystifying Mixed Outcomes of Self-Training: Pre-training Analyses on Non-Toy LLMs (2026.findings-eacl)

Challenge:	Existing tabular data synthesis methods fail to account for cross-modal heterogeneity of real-world tables, where structured continuous and discrete attributes coexist with unstructured long-text columns.
Approach:	They propose a framework that synergistically trains an LLM-based text generator and a deep-learning-based non-textual generator to quantify cross-modal semantic alignment.
Outcome:	The proposed framework outperforms state-of-the-art frameworks in fidelity, diversity, and task utility.

Copied to clipboard

Yusuke Nakamura, Hirokazu Kiyomaru, Chaoran Liu, Shuhei Kurita, Daisuke Kawahara

Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference (P19-1)

Challenge:	Recent studies on self-training report seemingly contradictory outcomes.
Approach:	They use OLMo-2 models as non-toy LLMs and perform multiple rounds of continual pre-training using self-generated text with different prompting strategies and data filtering.
Outcome:	The proposed model collapse is inherent to the training procedure itself, while self-improvement is likely owes its success to human-designed, strategic synthetic pipelines that inject external intelligence.

Copied to clipboard

Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, Iryna Gurevych

Textual Inference

Challenge:	Recent advances on abstractive summarization have led to fluent summaries, but factual errors in generated summary still severely limit their use in practice.
Approach:	They evaluate summaries produced by state-of-the-art models via crowdsourcing and show that factual errors occur frequently.
Outcome:	The proposed models can detect errors and reduce them by reranking alternative summaries.

Supporting Medical Relation Extraction via Causality-Pruned Semantic Dependency Forest (2022.coling-1)

Copied to clipboard

Yifan Jin, Jiangmeng Li, Zheng Lian, Chengbo Jiao, Xiaohui Hu

Learning Global Controller in Latent Space for Parameter-Efficient Fine-Tuning (2024.acl-long)

Challenge:	Medical relation extraction (MRE) tasks aims to extract relations between entities in medical literature.
Approach:	They propose to combine semantic and syntactic information from medical texts by using causal explanation theory.
Outcome:	Empirically, the proposed model outperforms existing methods on benchmark medical datasets.

Copied to clipboard

Zeqi Tan, Yongliang Shen, Xiaoxia Cheng, Chang Zong, Wenqi Zhang, Jian Shao, Weiming Lu, Yueting Zhuang

Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models (2022.findings-acl)

Challenge:	Large language models (LLMs) have shown remarkable performance, but their training costs are exorbitant.
Approach:	They propose a parameter-efficient method for exploring optimal solutions within latent space by using latent units to extract input representations from LLMs.
Outcome:	The proposed method improves performance on a range of natural language processing tasks.

Copied to clipboard

Robert Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, Sebastian Riedel

Causal-Debias: Unifying Debiasing in Pretrained Language Models and Fine-tuning via Causal Invariant Learning (2023.acl-long)

Challenge:	Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning.
Approach:	They propose to fine tune masked language models with training examples and task descriptions to reduce prompt engineering by using null prompts.
Outcome:	The proposed prompts can be used to improve few-shot learning by finetuning only the bias terms while updating only 0.1% of the parameters.

Copied to clipboard

Fan Zhou, Yuzhou Mao, Liu Yu, Yi Yang, Ting Zhong

AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Models (2022.findings-emnlp)

Challenge:	Existing methods to remove unwanted stereotypical associations from pretrained language models (PLMs) are often focused on removing unwanted stereotypes from PLMs.
Approach:	They propose a framework to remove unwanted stereotypical associations in pretrained language models . they propose bias-relevant factors are causal, while labelrelevant factors causal .
Outcome:	The proposed framework reduces stereotypical associations after PLMs are fine-tuned . the proposed framework mitigates bias from a causal invariant perspective .

Copied to clipboard

Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeongwook Kim, Jung-Woo Ha, Nako Sung, Dongsoo Lee

PATS: Sensitivity-aware Noisy Learning for Pretrained Language Models (2022.emnlp-main)

Challenge:	Existing approaches to improve inference efficiency by accelerating model fine-tuning have not been thoroughly explored.
Approach:	They propose to combine parameter-efficient adaptation and model compression to accelerate model . they propose to freeze binary parameters and scale scaling factors for target tasks .
Outcome:	The proposed algorithm achieves >10x compression ratio under 4-bit quantization and >1,000x reduction in trainable parameters.

Copied to clipboard

Yupeng Zhang, Hongzhi Zhang, Sirui Wang, Wei Wu, Zhoujun Li

A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains (2024.acl-long)

Challenge:	A wide range of NLP tasks benefit from fine-tuning of pretrained language models (PLMs), however, a number of redundant parameters which contribute less to the downstream task are observed in a directly fine- tuned model.
Approach:	They propose a noisy training mechanism which considers each parameter’s importance in the downstream task to help fine-tune pretrained language models.
Outcome:	The proposed method can be used to fine-tune pretrained language models on a wide range of tasks and consistently achieve higher performance.

Copied to clipboard

Alon Jacovi, Yonatan Bitton, Bernd Bohnet, Jonathan Herzig, Or Honovich, Michael Tseng, Michael Collins, Roee Aharoni, Mor Geva

Revealing the Importance of Semantic Retrieval for Machine Reading at Scale (D19-1)

Challenge:	Recent literature discusses automatic methods to evaluate reasoning to improve their correctness, but no fine-grained step-level datasets are available to enable thorough evaluation of such verification methods.
Approach:	They propose to benchmark automatic verifiers of complex Chain-of-Thought reasoning in open-domain question-answering settings using a dataset that includes comprehensive labels for relevance, attribution to evidence passages, and logical correctness of each reasoning step.
Outcome:	The proposed dataset shows that verifiers struggle at verifying reasoning chains, particularly verifying logical correctness and detecting contradictions.

Copied to clipboard

Yixin Nie, Songhe Wang, Mohit Bansal

Information Retrieval

Challenge:	Recent advances in representation learning have separated progress in both IR and MC . few studies have examined the relationship between retrieval and comprehension at different levels of granularity for development of MRS systems.
Approach:	They propose a simple yet effective pipeline system with consideration on hierarchical semantic retrieval at both paragraph and sentence level and their potential effects on the downstream task.
Outcome:	The proposed system achieves state-of-the-art on the leaderboard test sets of both FEVER and HOTPOTQA.

McPhraSy: Multi-Context Phrase Similarity and Clustering (2022.findings-emnlp)

Copied to clipboard

Amir Cohen, Hila Gonen, Ori Shapira, Ran Levy, Yoav Goldberg

Text Clustering

Challenge:	Existing methods for estimating phrase similarity use the phrase context only during training, instead relying on the phrase itself.
Approach:	They propose a novel algorithm that leverages multiple contexts during inference to estimate the similarity of phrases based on multiple context.
Outcome:	The proposed method outperforms existing models on two phrase similarity datasets by 13.3% and a new task that relies on phrase similarities in the product reviews domain.

MeNTi: Bridging Medical Calculator and LLM Agent with Nested Tool Calling (2025.naacl-long)

Copied to clipboard

Yakun Zhu, Shaohang Wei, Xu Wang, Kui Xue, Shaoting Zhang, Xiaofan Zhang

TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection (2022.emnlp-main)

Challenge:	Large Language Models (LLMs) have been widely used in medicine but are limited in their ability to fully address the complexities of the real world.
Approach:	They propose a universal agent architecture for Large Language Models that integrates a specialized medical toolkit and employs meta-tool and nested calling mechanisms to enhance LLM tool utilization.
Outcome:	The proposed framework improves the accuracy and performance of medical calculators in complex medical scenarios.

Copied to clipboard

Chaoya Jiang, Haiyang Xu, Chenliang Li, Ming Yan, Wei Ye, Shikun Zhang, Bin Bi, Songfang Huang

Visual Data in NLP

Challenge:	Existing vision-and-language pre-training models suffer from long visual sequences . experimental results show that TRIPS gains a speedup of 40% over previous similar VLP models .
Approach:	They propose an efficient vision-and-language pre-training model with text-relevant image patch selection, TRIPS, which reduces the visual sequence progressively with a text-guided patch-selection layer in the visual backbone for efficient training and inference.
Outcome:	The proposed model can speed up training and inference by 40% over previous models.

Hallucination Detox: Sensitivity Dropout (SenD) for Large Language Model Training (2025.acl-long)

Copied to clipboard

Shahrad Mohammadzadeh, Juan David Guerra, Marco Bonizzato, Reihaneh Rabbany, Golnoosh Farnadi

Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Texts (2024.naacl-long)

Challenge:	Existing studies have focused on identifying and addressing hallucinations in large language models (LLMs), but the impact of the training process on hallucinosity remains underexplored.
Approach:	They propose a training protocol to reduce hallucination variance by dropping embedding indices with significant variability and an unsupervised halluciation detection metric, Efficient EigenScore.
Outcome:	The proposed training protocol reduces hallucination variance during training by dropping embedding indices with significant variability.

Copied to clipboard

Sai Ashish Somayajula, Youwei Liang, Li Zhang, Abhishek Singh, Pengtao Xie

Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models (2023.findings-emnlp)

Challenge:	Pretrained language models have advanced natural language processing tasks significantly, but finetuning them on low-resource datasets presents significant challenges such as instability and overfitting.
Approach:	They propose a regularization method based on attention-guided weight mixup for finetuning PLMs on low-resource datasets.
Outcome:	The proposed method improves generalization and combats overfitting on two splits of the training dataset.

Copied to clipboard

Amirhossein Kazemnejad, Mehdi Rezagholizadeh, Prasanna Parthasarathi, Sarath Chandar

A Span-level Bidirectional Network for Aspect Sentiment Triplet Extraction (2022.emnlp-main)

Challenge:	Recent research has demonstrated that pre-trained language models acquire a broad range of knowledge about linguistic structures, encyclopedic relations, levels of commonsense, and even coding and reasoning rules.
Approach:	They propose a systematic framework to measure parametric knowledge utilization in pre-trained language models by extracting parametric information from a PLM and constructing a downstream task around this extracted knowledge.
Outcome:	The proposed framework extracts parametric knowledge from a PLM and constructs a downstream task around this extracted knowledge.

Copied to clipboard

Yuqi Chen, Chen Keming, Xian Sun, Zequn Zhang

Sentiment Analysis

Challenge:	Aspect Sentiment Triplet Extraction (ASTE) is a new fine-grained sentiment analysis task . recent studies have focused on solving aspects term extraction, opinion term extraction and aspect-level sentiment classification tasks individually or in combination of two subtasks.
Approach:	They propose a span-level bidirectional network which utilizes all possible spans as input and extracts triplets from spans bidirectionally.
Outcome:	The proposed framework outperforms state-of-the-art methods and improves performance . it can extract triplets of aspect terms, sentiments, and opinion terms from review sentences .

Dynamically Disentangling Social Bias from Task-Oriented Representations with Adversarial Attack (2021.naacl-main)

Copied to clipboard

Liwen Wang, Yuanmeng Yan, Keqing He, Yanan Wu, Weiran Xu

SiBert: Enhanced Chinese Pre-trained Language Model with Sentence Insertion (2020.lrec-1)

Challenge:	Existing methods to learn representations from text often reflect social biases . previous methods rely on pre-specified direction or suffer from unstable training .
Approach:	They propose an adversarial disentangled debiasing model to decouple social bias attributes from intermediate representations trained on the main task.
Outcome:	The proposed model decouples social bias attributes from intermediate representations trained on the main task.

Copied to clipboard

Jiahao Chen, Chenjie Cao, Xiuyan Jiang

On Transferability of Bias Mitigation Effects in Language Model Fine-Tuning (2021.naacl-main)

Challenge:	Recent studies show that pre-trained models can learn unsupervised language representations by self-supervised tasks on large-scale corpora.
Approach:	They propose a pre-training task called Sentence Insertion for Chinese query-passage pairs NLP tasks . they propose 'word segmentation' method to enhance Chinese Bert performance .
Outcome:	The proposed task improves Chinese pre-trained models significantly.

Copied to clipboard

Xisen Jin, Francesco Barbieri, Brendan Kennedy, Aida Mostafazadeh Davani, Leonardo Neves, Xiang Ren

BERT Busters: Outlier Dimensions that Disrupt Transformers (2021.findings-acl)

Challenge:	PTLMs can exhibit biases against protected groups in a host of modeling tasks . but, fine-tuned LMs may propagate bias to downstream classifiers .
Approach:	They propose to use upstream bias mitigation techniques to reduce bias on downstream tasks by fine-tuning an upstream model and applying it to a downstream model.
Outcome:	The proposed model reduces bias on hate speech detection, toxicity detection and coreference resolution tasks over bias factors.

Copied to clipboard

Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, Anna Rumshisky

MUX-PLMs: Data Multiplexing for High-throughput Language Models (2023.findings-emnlp)

Challenge:	Existing studies show that pre-trained Transformers are remarkably robust to pruning.
Approach:	They show that pre-trained Transformer encoders are surprisingly fragile to pruning . they show that disabling them significantly degrades both the MLM loss and the downstream task performance.
Outcome:	The results show that the removal of features in pre-trained transformers significantly degrades both the MLM loss and the downstream task performance.

Copied to clipboard

Vishvak Murahari, Ameet Deshpande, Carlos Jimenez, Izhak Shafran, Mingqiu Wang, Yuan Cao, Karthik Narasimhan

FlauBERT: Unsupervised Language Model Pre-training for French (2020.lrec-1)

Challenge:	MUX-PLMs are high-throughput pre-trained language models that can be fine-tuned for any downstream task to yield high-performance.
Approach:	They propose to train language models with data multiplexing to achieve 2x/5x inference speedup . they use multiplexers to entangle and disentangle inputs to achieve the same performance .
Outcome:	MUX-PLMs achieve 2x/5x inference speedup with 1-4 % drop on broad suite of tasks.

Copied to clipboard

Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, Didier Schwab

Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pre-training (2026.findings-eacl)

Challenge:	Language models are a key step to achieve state-of-the-art results in many different Natural Language Processing (NLP) tasks.
Approach:	They propose to use a language model that is pre-trained on a large and heterogeneous French corpus to train continuous word representations.
Outcome:	The proposed model outperforms existing models on a large and heterogeneous French corpus.

Copied to clipboard

Jeffrey Li, Joshua P Gardner, Doug Kang, Fangping Shi, Karanjeet Singh, Chun-Liang Li, Herumb Shandilya, David Leo Wright Hall, Oncel Tuzel, Percy Liang, Ludwig Schmidt, Hadi Pouransari, Fartash Faghri

Federated Learning for Spoken Language Understanding (2020.coling-main)

Challenge:	Existing open-source datasets predominantly apply a single fixed extractor to all webpages.
Approach:	They propose to take a Union over different extractors to improve model performance . they show that extractor choice can significantly impact downstream task performance based on content type .
Outcome:	The proposed approach can increase the token yield of DCLM-Baseline by 71% while maintaining benchmark performance.

Copied to clipboard

Zhiqi Huang, Fenglin Liu, Yuexian Zou

Evaluating Parameter Efficient Learning for Generation (2022.emnlp-main)

Challenge:	Existing methods to improve robustness of models focus on a single dataset . but, there are few studies on how to combine merits of different datasets .
Approach:	They propose a federated learning framework that could unify datasets and tasks . they propose MV-Encoder as backbone of the framework to provide multi-granularity text representations .
Outcome:	The proposed framework improves on two SLU benchmark datasets and federated learning settings.

Copied to clipboard

Peng Xu, Mostofa Patwary, Shrimai Prabhumoye, Virginia Adams, Ryan Prenger, Wei Ping, Nayeon Lee, Mohammad Shoeybi, Bryan Catanzaro

Task-aware Block Pruning with Output Distribution Signals for Large Language Models (2026.findings-eacl)

Challenge:	Parameter efficient learning methods (PERMs) are gaining attention for their ability to adapt to a downstream task.
Approach:	They propose to use parameter efficient learning methods to improve model adaptation . they compare in-domain evaluations and generalizations to unseen domains and new datasets .
Outcome:	The proposed method outperforms finetuning and PERMs in in-domain evaluations.

Copied to clipboard

Song-ha Jo, Youngrok Ko, Sang-goo Lee, Jinseok Seol

The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse (2024.findings-acl)

Challenge:	Existing methods to estimate block importance rely on representation similarity or computationally expensive sensitivity analyses to estimate task-aware model behavior.
Approach:	They propose a novel approach that quantifies block-level uncertainty from the statistics of each block’s early-exited output distribution on a calibration dataset.
Outcome:	Experiments show that the proposed approach preserves downstream task performance while reducing inference latency and computational cost.

Copied to clipboard

Wanli Yang, Fei Sun, Xinyu Ma, Xun Liu, Dawei Yin, Xueqi Cheng

TasTe: Teaching Large Language Models to Translate through Self-Reflection (2024.acl-long)

Challenge:	Even a single edit can trigger model collapse, manifesting as significant performance degradation in various benchmark tasks.
Approach:	They propose to use perplexity as a surrogate metric to determine whether an edited model's performance is affected by a single edit.
Outcome:	The proposed method shows that even a single edit can cause model collapse, manifesting as significant performance degradation in various benchmark tasks.

Copied to clipboard

Yutong Wang, Jiali Zeng, Xuebo Liu, Fandong Meng, Jie Zhou, Min Zhang

CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling (2024.emnlp-main)

Challenge:	Existing approaches to enhance LLMs' performance in machine translation are unable to fully exploit their instruction-following capabilities.
Approach:	They propose a framework for translating through self-reflection that involves two stages of inference . they propose to use the framework to refine LLMs' preliminary translations .
Outcome:	The proposed framework can produce translation outputs that match the quality of NMT systems.

Copied to clipboard

Yu Bai, Xiyuan Zou, Heyan Huang, Sanxing Chen, Marc-Antoine Rondeau, Yang Gao, Jackie Cheung

Pass off Fish Eyes for Pearls: Attacking Model Selection of Pre-trained Models (2022.acl-long)

Challenge:	Recent studies have found that information relevant to the next token prediction task accumulates in the hidden representations of just a few tokens.
Approach:	They propose a method that integrates attention preferences useful for a downstream task into the eviction process of hidden states.
Outcome:	The proposed method performs better on comprehension and retrieval tasks while preserving language modeling perplexity.

Copied to clipboard

Biru Zhu, Yujia Qin, Fanchao Qi, Yangdong Deng, Zhiyuan Liu, Maosong Sun, Ming Gu

The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models (2023.findings-emnlp)

Challenge:	Existing feature-based model selection methods are vulnerable to fine-tuning and are not reliable indicators for the PTM’s transferability.
Approach:	They propose feature-based model selection methods which assess PTMs’ transferability to a specific task in a fast way without fine-tuning.
Outcome:	The proposed methods can make FMS mistakenly judge transferability of models and can be combined with the backdoor attack to misguide the FMS to select poisoned models.

Copied to clipboard

Satya Sai Srinath Namburi, Makesh Sreedhar, Srinath Srinivasan, Frederic Sala

Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2022.naacl-main)

Challenge:	Existing research on LLM compression focuses on general metrics like perplexity or downstream task accuracy.
Approach:	They propose to quantify the effect of pruning and quantization on model quality . they use the LAMA and LM-Harness benchmarks to quantify compression techniques .
Outcome:	The proposed compression techniques provide faster inference, smaller memory footprints, and enables local deployment.

Copied to clipboard

Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, Xiang Ren

Learn What Is Possible, Then Choose What Is Best: Disentangling One-To-Many Relations in Language Through Text-based Games (2022.findings-emnlp)

Challenge:	Pretrained language models are typically learned over a large, static corpus and fine-tuned for various downstream tasks.
Approach:	They propose to continuously update a pretrained language model to adapt to emerging data and to keep track of the model's performance.
Outcome:	The proposed model can adapt to new corpora while retaining knowledge in earlier domains.

Copied to clipboard

Benjamin Towle, Ke Zhou

Unsupervised Label Refinement Improves Dataless Text Classification (2021.findings-acl)

Challenge:	Language models pre-trained on large self-supervised corpora, followed by task-specific fine-tuning has become the dominant paradigm in NLP.
Approach:	They propose to train language models pre-trained on large self-supervised corpora, followed by task-specific fine-tuning on the target domain.
Outcome:	The proposed model improves on the previous state-of-the-art model on the Jericho Walkthroughs dataset by 49%.

Copied to clipboard

Zewei Chu, Karl Stratos, Kevin Gimpel

Zero-Shot Cross-Lingual Transfer with Meta Learning (2020.emnlp-main)

Challenge:	Prior work has shown that dataless classifiers can classify text into categories without using annotated training data.
Approach:	They propose a clustering-based approach that assigns a score to any document paired with a label description.
Outcome:	The proposed method improves classification performance across multiple datasets and makes it more robust to label descriptions.

Copied to clipboard

Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, Isabelle Augenstein

Low-Resource NLP

Challenge:	There are more than 7,000 languages spoken in the world, over 90 of which have more than 10 million native speakers each.
Approach:	They propose to use meta-learning to train a model on multiple languages at the same time . they use standard supervised, zero-shot cross-lingual, and few-shot crosses-lingual settings for different natural language understanding tasks.
Outcome:	The proposed setup improves on the state-of-the-art for a total of 15 languages.

Shielded Representations: Protecting Sensitive Attributes Through Iterative Gradient-Based Projection (2023.findings-acl)

Copied to clipboard

Shadi Iskander, Kira Radinsky, Yonatan Belinkov

KALA: Knowledge-Augmented Language Model Adaptation (2022.naacl-main)

Challenge:	Natural language processing models tend to learn and encode social biases present in the data.
Approach:	They propose a method for removing non-linear encoded concepts from neural representations by iteratively training neural classifiers to predict a particular attribute, followed by a projection of the representation on a hypersurface.
Outcome:	The proposed method removes non-linear encoded concepts from neural representations.

Copied to clipboard

Minki Kang, Jinheon Baek, Sung Ju Hwang

A Context-Aware Approach for Enhancing Data Imputation with Pre-trained Language Models (2025.coling-main)

Challenge:	Pre-trained language models (PLMs) have proved to be effective on various natural language understanding tasks.
Approach:	They propose a domain adaption framework which modulates the intermediate hidden representations of PLMs with domain knowledge, consisting of entities and their relational facts.
Outcome:	The proposed framework outperforms adaptive pre-training on question answering and named entity recognition tasks on multiple datasets across different domains.

Copied to clipboard

Ahatsham Hayat, Mohammad R. Hasan

On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model (2022.naacl-main)

Challenge:	Existing approaches to handle missing data in tabular datasets rely on numerical estimations instead of pre-trained language models (LMs).
Approach:	They propose a method that leverages pre-trained language models to create contextually relevant descriptors for missing values.
Outcome:	The proposed approach outperforms the best-performing baselines in MCAR, MAR, and MNAR scenarios and offers a cost-effective solution for resource-constrained environments.

Copied to clipboard

Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Woo Ha, Nako Sung

Cross-lingual Contextualized Phrase Retrieval (2024.findings-emnlp)

Challenge:	Recent studies on large-scale in-context language models have reported successful in-const zero- and few-shot learning ability.
Approach:	They investigate the effects of the pretraining corpus on in-context learning in a Korean-centric model.
Outcome:	The study shows that pretraining corpus size does not determine in-context learning ability . the findings suggest that in-constext learning is not always competitive .

Copied to clipboard

Huayang Li, Deng Cai, Zhi Qu, Qu Cui, Hidetaka Kamigaito, Lemao Liu, Taro Watanabe

AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning (2022.emnlp-main)

Challenge:	Phrase-level dense retrieval has shown many appealing characteristics in downstream NLP tasks.
Approach:	They propose a task formulation of dense retrieval, cross-lingual contextualized phrase retrieval . they extract pairs of cross-linguistic phrases using word alignment information .
Outcome:	The proposed task formulation surpasses baselines on the phrase retrieval task and a downstream task, i.e., machine translation, and achieves top-1 accuracy 13 points higher.

Copied to clipboard

Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao

CareCorpus+: Expanding and Augmenting Caregiver Strategy Data to Support Pediatric Rehabilitation (2024.emnlp-main)

Challenge:	Standard fine-tuning of large pre-trained language models requires updating hundreds of millions to billions of parameters and storing a large copy of the PLM weights for every task.
Approach:	They propose a parameter-efficient fine-tuning technique where small trainable components are injected into the PLM and updated during fine-uning.
Outcome:	The proposed method outperforms SOTA parameter-efficient fine-tuning and full model fine-uning on GLUE development set with RoBERTa-large encoder.

Copied to clipboard

Shahla Farzana, Ivana Lucero, Vivian Villegas, Vera Kaelin, Mary Khetani, Natalie Parde

Challenge:	Existing studies on caregiver strategy classification in pediatric rehabilitation contexts are under-resourced and under-studied.
Approach:	They propose to manually categorized 4,037 caregiver strategies in a pediatric rehabilitation setting, and manually supplement target task data with publicly relevant child health forums.
Outcome:	The proposed method improves the quality of the dataset and shows that it performs well.

Applying Natural Annotation and Curriculum Learning to Named Entity Recognition for Under-Resourced Languages (2022.coling-1)

Copied to clipboard

Valeriy Lobov, Alexandra Ivoylova, Serge Sharoff

Analyzing Redundancy in Pretrained Transformer Models (2020.emnlp-main)

Challenge:	Existing approaches to build NLP models for low-resourced languages rely on machine translation or cross-lingual transfer.
Approach:	They propose to use natural annotations to build synthetic training sets from resources not originally designed for the target downstream task.
Outcome:	The proposed model achieves the F1 score of 0.78 for Belarusian starting from zero resources compared to the baseline of 0.63 for English . the proposed model can be fine-tuned to reflect linguistic properties, such as the grammatical case and gender, for the Slavic languages.

Copied to clipboard

Fahim Dalvi, Hassan Sajjad, Nadir Durrani, Yonatan Belinkov

Task-adaptive Pre-training of Language Models with Word Embedding Regularization (2021.findings-acl)

Challenge:	Recent work shows that transformer-based deep NLP models are over-parameterized and do not require all the representational power lent by the rich architectural choices during inference.
Approach:	They define a notion of Redundancy and propose a feature-based transfer learning procedure which maintains 97% performance while using at-most 10% of the original neurons.
Outcome:	The proposed model maintains 97% performance while using 10% of the original neurons.

Copied to clipboard

Kosuke Nishida, Kyosuke Nishida, Sen Yoshida

SCRIPT: Self-Critic PreTraining of Transformers (2021.naacl-main)

Challenge:	Pre-trained language models acquire domain-independent knowledge through pre-training with massive textual resources.
Approach:	They propose a task-adaptive pre-training process that makes static embeddings close to the word embedds obtained in the target domain.
Outcome:	The proposed process improves on BioASQ and SQuAD when the pre-training corpora were not dominated by indomain data.

Copied to clipboard

Erik Nijkamp, Bo Pang, Ying Nian Wu, Caiming Xiong

Multi-Relational Script Learning for Discourse Relations (P19-1)

Challenge:	Existing methods for representation learning of text are masked language modeling (MLM) a language model is trained to learn universal contextual embeddings, which are fine-tuned on a down-stream task.
Approach:	They propose a self-critic pretraining transformer for representation learning of text . they demonstrate improved sample-efficiency and improved performance over strong baselines .
Outcome:	The proposed model improves sample-efficiency and performance over strong baselines.

Copied to clipboard

I-Ta Lee, Dan Goldwasser

Uncovering Latent Arguments in Social Media Messaging by Employing LLMs-in-the-Loop Strategy (2025.findings-naacl)

Challenge:	Existing script knowledge models only represent a single event relationship, co-occurrence . this is coarse for commonsense, which should account for fine-grained relationships .
Approach:	They propose to view learning event embedding as a multi-relational problem . they model a rich set of event relations derived from the Penn Discourse Tree Bank .
Outcome:	The proposed model captures different aspects of event pairs, including cause and contrast.

Copied to clipboard

Tunazzina Islam, Dan Goldwasser

Challenge:	Supervised methods are adept at text categorization, but dynamic nature of social media debates pose challenges for them . traditional methods for extracting themes from public discourse often reveal overarching patterns that might not capture specific nuances.
Approach:	They propose a generic approach that leverages the advanced capabilities of Large Language Models to extract latent arguments from social media messaging.
Outcome:	The proposed approach leverages the advanced capabilities of Large Language Models (LLMs) to extract latent arguments from social media messaging.

How did we get here? Summarizing conversation dynamics (2024.naacl-long)

Copied to clipboard

Yilun Hua, Nicholas Chernogor, Yuzhe Gu, Seoyeon Jeong, Miranda Luo, Cristian Danescu-Niculescu-Mizil

Depth-Wise Attention (DWAtt): A Layer Fusion Method for Data-Efficient Classification (2024.lrec-main)

Challenge:	Throughout a conversation, the way participants interact with each other is in constant flux.
Approach:	They propose to summarize conversations by constructing human-written summaries and exploring automated baselines.
Outcome:	The summarizing tools help both humans and automated systems forecast toxic behavior in conversations.

Copied to clipboard

Muhammad ElNokrashy, Badr AlKhamissi, Mona Diab

Detecting Loanwords in Emakhuwa: An Extremely Low-Resource Bantu Language Exhibiting Significant Borrowing from Portuguese (2024.lrec-main)

Challenge:	Language Models pretrained on large textual data can encode different types of knowledge simultaneously.
Approach:	They propose a method to re-surface intermediate layer features from non-final layers by combining them with a concatenation-based layer fusion method.
Outcome:	The proposed method outperforms the baseline model on large datasets and shows 3.68 9.73% gain.

Copied to clipboard

Felermino Dario Mario Ali, Henrique Lopes Cardoso, Rui Sousa-Silva

How does BERT’s attention change when you fine-tune? An analysis methodology and a case study in negation scope (2020.acl-main)

Challenge:	Existing corpora in African languages reveal significant spelling inconsistencies, contributing to poor-quality textual data when encountered in written form.
Approach:	They propose a supervised method to identify loanwords in Portuguese . they employ traditional machine learning algorithms incorporating handcrafted features .
Outcome:	The proposed method achieves the F1-score of 93% in Emakhuwa, borrowed from Portuguese.

Copied to clipboard

Yiyun Zhao, Steven Bethard

MUSCLE: A Model Update Strategy for Compatible LLM Evolution (2024.findings-emnlp)

Challenge:	Recent work probing pre-trained language models for downstream tasks is difficult to explain . a growing body of research is devoted to understanding what linguistic properties these language models have acquired.
Approach:	They propose a procedure and analysis method that takes a hypothesis of how a transformer-based model might encode a linguistic phenomenon and tests its validity.
Outcome:	The proposed method tests a hypothesis that some attention heads will consistently attend from a word in negation scope to the negation cue.

Copied to clipboard

Jessica Echterhoff, Fartash Faghri, Raviteja Vemulapalli, Ting-Yao Hu, Chun-Liang Li, Oncel Tuzel, Hadi Pouransari

Adapting Monolingual Models: Data can be Scarce when Language Similarity is High (2021.findings-acl)

Challenge:	Large Language Models (LLMs) are regularly updated to enhance performance, typically through changes in data or architecture.
Approach:	They propose a method to minimize the extent of instance regression in model updates by training a compatibility adapter that can enhance task fine-tuned language models.
Outcome:	The proposed approach reduces regressions by up to 40% when updating LLMs to newer versions while maintaining overall performance gains.

Copied to clipboard

Wietse de Vries, Martijn Bartelds, Malvina Nissim, Martijn Wieling

FeatureBART: Feature Based Sequence-to-Sequence Pre-Training for Low-Resource NMT (2022.coling-1)

Challenge:	Large pre-trained language models are the dominant approach for solving many tasks in natural language processing.
Approach:	They propose to retrain the lexical layers of four BERT-based models using data from two low-resource target languages while the Transformer layers are independently finetuned on a POS-tagging task in the model's source language.
Outcome:	The proposed method achieves high performance for both target and target languages with high similarity.

Copied to clipboard

Abhisek Chakrabarty, Raj Dabre, Chenchen Ding, Hideki Tanaka, Masao Utiyama, Eiichiro Sumita

Hard Prompts Made Interpretable: Sparse Entropy Regularization for Prompt Tuning with RL (2024.acl-long)

Challenge:	FeatureBART is a linguistically motivated sequence-to-sequence monolingual pre-training strategy . syntactic features such as lemma, part-of-speech and dependency labels are incorporated into the pre-trained model .
Approach:	They propose a linguistically motivated sequence-to-sequence monolingual pre-training strategy that incorporates syntactic features into the framework.
Outcome:	The proposed model improves translation quality in bilingual and multilingual settings over models that do not use features.

Copied to clipboard

Yunseon Choi, Sangmin Bae, Seonghyun Ban, Minchan Jeong, Chuheng Zhang, Lei Song, Li Zhao, Jiang Bian, Kee-Eung Kim

LLMs Deceive Unintentionally: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactions (2026.findings-acl)

Challenge:	Prompt tuning is an important technique for directing model behaviors and eliciting desired responses.
Approach:	They propose to find optimal prompt tokens using soft Q-learning to optimize models for prompt tuning.
Outcome:	The proposed method improves on baseline prompt tuning, and the results are more natural and interpretable.

Copied to clipboard

Xuhao Hu, Wang Peng, Xiaoya Lu, Dongrui Liu, Xuanjing Huang, Jing Shao

I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors (2023.findings-acl)

Challenge:	Existing studies have shown that LLMs finetuned on incorrect completions can exhibit harmful behaviors, which is called emergent misalignment.
Approach:	They investigate whether LLMs finetuned on incorrect completions can exhibit harmful behaviors . they find that 1% of misalignment data is sufficient to decrease honest behavior .
Outcome:	The proposed model can be misaligned on errors within narrow domains to exhibit harmful behaviors . the proposed model is able to exhibit dishonest behavior with only 10% biased user population .

Copied to clipboard

Tuhin Chakrabarty, Arkadiy Saakyan, Olivia Winn, Artemis Panagopoulou, Yue Yang, Marianna Apidianaki, Smaranda Muresan

Visual Data in NLP

Challenge:	Visual metaphors are powerful rhetorical devices used to communicate creative ideas through images.
Approach:	They propose to generate visual metaphors from linguistic metaphors by using large language models and Diffusion models.
Outcome:	The proposed task requires the ability to model implicit meaning and compositionality.

Close or Cloze? Assessing the Robustness of Large Language Models to Adversarial Perturbations via Word Recovery (2025.coling-main)

Copied to clipboard

Luke Moffett, Bhuwan Dhingra

UOR: Universal Backdoor Attacks on Pre-trained Language Models (2024.findings-acl)

Challenge:	Existing models implicitly recover the original text, but it is unclear when they rely on context and when they implicitly do so.
Approach:	They propose to use a dictionary to recover adversarial words by using a phonetic, typo, and visual attack to study word recovery performance.
Outcome:	The proposed model outperforms open-source models on hateful, offensive, and toxic classification tasks.

Copied to clipboard

Wei Du, Peixuan Li, Haodong Zhao, Tianjie Ju, Ge Ren, Gongshen Liu

Information Parity: Measuring and Predicting the Multilingual Capabilities of Language Models (2024.findings-emnlp)

Challenge:	Existing methods to attack pre-trained language models rely on manual selection of triggers and backdoor representations.
Approach:	They propose a backdoor attack method that turns manual selection into automatic optimization . they propose to use poisoned contrastive learning to learn more uniform backdoor representations .
Outcome:	The proposed method achieves better attack performance on text classification tasks compared to manual methods.

Copied to clipboard

Alexander Tsvetkov, Alon Kipnis

Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study (2023.emnlp-main)

Challenge:	Large Language Models (LLMs) are increasingly used in user-facing applications worldwide, necessitating handling multiple languages across various tasks.
Approach:	They propose a metric called Information Parity (IP) that can predict an LLM’s capabilities across multiple languages in a task-agnostic manner.
Outcome:	The proposed metric can predict LLM’s capabilities across multiple languages in a task-agnostic manner.

Copied to clipboard

Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, Anima Anandkumar, Bryan Catanzaro

Interactive Refinement of Cross-Lingual Word Embeddings (2020.emnlp-main)

Challenge:	a recent study shows that retrieval-augmented LMs can improve text generation quality and accuracy.
Approach:	They propose a model that reproduces RETRO parameters while retrieving a text corpus . they find RETRO outperforms GPT on text generation with less repetition .
Outcome:	The proposed model outperforms standard retrieval-augmented GPT and retrieval augmented GTP on text generation and accuracy tasks.

Copied to clipboard

Michelle Yuan, Mozhi Zhang, Benjamin Van Durme, Leah Findlater, Jordan Boyd-Graber

BLIND: Bias Removal With No Demographics (2023.acl-long)

Challenge:	Cross-lingual word embeddings transfer knowledge between languages to models trained on resource-rich languages can predict in low-resource languages.
Approach:	They propose an interactive system to quickly refine cross-lingual word embeddings for a given classification problem.
Outcome:	The proposed system improves on identifying health-related text in four low-resource languages.

Copied to clipboard

Hadas Orgad, Yonatan Belinkov

Information Retrieval

Challenge:	Numerous methods to mitigate social biases require prior knowledge of the demographics in the dataset, such as gender or race.
Approach:	They propose a method for bias removal without prior knowledge of demographics in the dataset.
Outcome:	Experiments with racial and gender biases in sentiment classification and occupation classification tasks show that BLIND mitigates biase . BLINT is competitive with methods that require demographic information and sometimes surpasses them.

Multilingual Data Filtering using Synthetic Data from Large Language Models (2025.findings-emnlp)

Copied to clipboard

Jonas Waldendorf, Barry Haddow, Alexandra Birch, Mateusz Klimaszewski

Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefix (2023.findings-emnlp)

Challenge:	Recent studies have shown that effective filters can be created by utilising Large Language Models to synthetically label data, which is then used to train smaller neural models for filtering purposes.
Approach:	They extend this approach to languages beyond English to train neural models for filtering purposes.
Outcome:	The proposed approach is effective at filtering parallel text for translation quality and filtering for domain specificity.

Copied to clipboard

Kuan-Hao Huang, Liang Tan, Rui Hou, Sinong Wang, Amjad Almahairi, Ruty Rinott

CHIRON: Rich Character Representations in Long-Form Narratives (2024.findings-emnlp)

Challenge:	a large pre-trained language model can cause computational burdens in inference time due to multiple forward passes.
Approach:	They propose a method to learn fixed text representations with source tasks . they learn a task-specific prefix for each source task independently and combine them .
Outcome:	The proposed method improves generalizability of representations with source tasks.

Copied to clipboard

Alexander Gurung, Mirella Lapata

CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts (2025.naacl-long)

Challenge:	Existing systems for character representation have simplified the problem of representing complex characters via graphs and brief character descriptions.
Approach:	They propose a ‘character sheet’ based representation that organizes and filters textual information about characters.
Outcome:	The proposed representation organizes and filters textual information about characters and is better and more flexible than previous models.

Copied to clipboard

Zhenpeng Su, Xing W, Zijia Lin, Yizhe Xiong, Minxuan Lv, Guangyuan Ma, Hui Chen, Songlin Hu, Guiguang Ding

A Comprehensive Analysis of Preprocessing for Word Representation Learning in Affective Tasks (2020.acl-main)

Challenge:	Large language models (LLMs) have been attracting much attention due to their impressive performance in all kinds of downstream tasks.
Approach:	They propose a mix-of-experts model that allows the model size to grow without raising training costs.
Outcome:	The proposed model outperforms existing models in perplexity and robustness tests.

Copied to clipboard

Nastaran Babanejad, Ameeta Agrawal, Aijun An, Manos Papagelis

Sentiment Analysis

Challenge:	Affective tasks such as sentiment analysis, emotion classification and sarcasm detection have enjoyed great popularity in recent years.
Approach:	They conduct a comprehensive analysis of the role of preprocessing techniques in affective analysis based on word vector models.
Outcome:	The proposed model is the first of its kind and provides useful insights on the role of each preprocessing technique when applied at the training phase, commonly ignored in pretrained word vector models, and/or at the downstream task phase.

Scalable and Domain-General Abstractive Proposition Segmentation (2024.findings-emnlp)

Copied to clipboard

Mohammad Javad Hosseini, Yang Gao, Tim Baumgärtner, Alex Fabrikant, Reinald Kim Amplayo

Text Segmentation

Challenge:	Several recent studies have demonstrated the utility of proposition segmentation for downstream tasks.
Approach:	They propose a scalable, yet accurate, proposition segmentation model that can be supervised by LLMs.
Outcome:	The proposed model improves on training on annotated datasets and shows that it is easy to use.

CoBA: Counterbias Text Augmentation for Mitigating Various Spurious Correlations via Semantic Triples (2025.emnlp-main)

Copied to clipboard

Kyohoon Jin, Juhwan Choi, JungMin Yun, Junho Lee, Soojin Jang, YoungBin Kim

OpenKorPOS: Democratizing Korean Tokenization with Voting-Based Open Corpus Annotation (2022.lrec-1)

Challenge:	Spurious correlations are patterns that appear in datasets but do not represent genuine relationships.
Approach:	They propose a more general form of counterfactual data augmentation that tackles multiple biases . they propose 'CoBA' that decomposes text into subject-predicate-object triples and modifies them to disrupt spurious correlations.
Outcome:	The proposed framework reduces biases and strengthens out-of-distribution resilience.

Copied to clipboard

Sangwhan Moon, Won Ik Cho, Hye Joo Han, Naoaki Okazaki, Nam Soo Kim

Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models (2025.acl-long)

Challenge:	Korean uses spaces at larger-than-word boundaries, unlike other East-Asian languages.
Approach:	They propose to use Korean morphological analyzers to provide a sequence of morpheme-level tokens, losing information in the tokenization process.
Outcome:	The proposed scheme improves existing tagging scheme and makes it friendlier to generative tasks.

Copied to clipboard

Xinlin Zhuang, Jiahui Peng, Ren Ma, Yinfan Wang, Tianyi Bai, Xingjian Wei, Qiu Jiantao, Chi Zhang, Ying Qian, Conghui He

Are ELECTRA’s Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similarity (2024.findings-emnlp)

Challenge:	composition of pre-training datasets for large language models remains undisclosed . current methods for evaluating data quality are limited by single-dimensional evaluation or redundancy-focused strategies.
Approach:	They propose a multi-dimensional data selection method that integrates dimensions with existing quality metrics through learned optimal weightings.
Outcome:	The proposed method doubles convergence speed for 1.3B model models and improves downstream task performance by 3.23%.

Copied to clipboard

Ivan Rep, David Dukić, Jan Šnajder

Fair and Argumentative Language Modeling for Computational Argumentation (2022.acl-long)

Challenge:	ELECTRA's sentence embeddings are poorer than BERT's, resulting in a significant drop in performance for semantic textual similarity (STS).
Approach:	They propose to use a truncated model fine-tuning method to repair the embeddings by reducing the number of parameters and producing smaller embeddables.
Outcome:	The proposed method improves Spearman correlation coefficient by over 8 points while increasing parameter efficiency on the STS Benchmark.

Copied to clipboard

Carolin Holtermann, Anne Lauscher, Simone Ponzetto

Retrieval-augmented Video Encoding for Instructional Captioning (2023.findings-acl)

Challenge:	Recent work on stereotypical biases in semantic spaces is still in its infancy . we present a novel resource for bias measurement specifically tailored to argumentation .
Approach:	They propose a resource for bias measurement specifically tailored to argumentation . they use argumentative fine-tuning and debiasing to assess intrinsic bias .
Outcome:	The proposed approach is more sustainable and parameter-efficient than full fine-tuning . it can remove bias in general and argumentative language models while improving model performance in downstream tasks.

Copied to clipboard

Yeonjoon Jung, Minsoo Kim, Seungtaek Choi, Jihyuk Kim, Minji Seo, Seung-won Hwang

On the Impact of Calibration Data in Post-training Quantization and Pruning (2024.acl-long)

Challenge:	Instructional videos provide a detailed multimodal context of each procedure in instruction. key-object degeneracy is a problem for machine systems, causing incorrect captions.
Approach:	They propose a retrieval-based framework to augment the model representations in the presence of key-object degeneracy.
Outcome:	The proposed framework can be extended over baselines using modalities with key-object degeneracy.

Copied to clipboard

Miles Williams, Nikolaos Aletras

Tribrid: Stance Classification with Neural Inconsistency Detection (2021.emnlp-main)

Challenge:	Quantization and pruning are the foundations of compression for large language models . however, no prior work has investigated how calibration data impacts performance of compression methods.
Approach:	They propose an empirical study on the effect of calibration data on LLM performance.
Outcome:	The proposed methods improve performance in a post-training setting.

Copied to clipboard

Song Yang, Jacopo Urbani

Text Classification

Challenge:	a new neural architecture can be used to classify stances on social media without relying on linguistic features.
Approach:	They propose a neural architecture where the input also includes automatically generated negated perspectives over a given claim.
Outcome:	The proposed model improves on the original input and removes doubtful predictions over the retained information.

Few-Shot Natural Language to First-Order Logic Translation via Code Generation (2025.naacl-long)

Copied to clipboard

Junnan Liu

Text Generation

Challenge:	Recent studies have focused on translation of natural language to first-order logical formula (NL-FOL) but these methods face challenges such as inconsistency between training and inference phases and data-intensive finetuning process.
Approach:	They propose a method for translating natural language into first-order logical formulas using code snippets.
Outcome:	The proposed method surpasses training-free baselines and is comparable to supervised models trained on the full training data.

Learning Latent Trees with Stochastic Perturbations and Differentiable Dynamic Programming (P19-1)

Copied to clipboard

Caio Corro, Ivan Titov

Latent Factor Models Meets Instructions: Goal-conditioned Latent Factor Discovery without Task Supervision (2025.naacl-long)

Challenge:	Discrete structures are ubiquitous in the study of natural languages, for example in morphology, syntax and discourse analysis.
Approach:	They treat projective dependency trees as latent variables in their probabilistic model and induce them in such a way as to be beneficial for a downstream task.
Outcome:	The proposed model is effective on sentiment analysis and natural language inference tasks and is fully differentiable.

Copied to clipboard

Zhouhang Xie, Tushar Khot, Bhavana Dalvi Mishra, Harshit Surana, Julian McAuley, Peter Clark, Bodhisattwa Prasad Majumder

How Entangled is Factuality and Deception in German? (2024.findings-emnlp)

Challenge:	Instruction-following LLMs have recently allowed systems to discover hidden concepts from a collection of unstructured documents based on a natural language description of the purpose of the discovery (i.e., goal).
Approach:	They propose a goal-oriented latent factor discovery system that integrates LLM’s instruction-following ability with statistical models to handle large, noisy datasets where LLM reasoning alone falls short.
Outcome:	The proposed system improves task performance by 5-52% over baselines and 1.8 times as often as the best alternative, on average, in human evaluation.

Copied to clipboard

Aswathy Velutharambath, Amelie Wuehrl, Roman Klinger

Shallow Domain Adaptive Embeddings for Sentiment Analysis (D19-1)

Challenge:	Existing research on deception detection and fact checking conflates factual accuracy with truthfulness . a belief-based deception framework defines texts as deceptive when there is a mismatch between what people say and what they truly believe .
Approach:	They assess if presumed patterns of deception generalize to German language texts . they gauge the impact of deceptiveness on the downstream task of fact checking .
Outcome:	The proposed framework disentangles deception when there is a mismatch between what people say and what they truly believe . the proposed framework does not find any correlation with established cues of deception .

Copied to clipboard

Prathusha K Sarma, Yingyu Liang, William Sethares

CHiLL: Zero-shot Custom Interpretable Feature Extraction from Clinical Notes with Large Language Models (2023.findings-emnlp)

Challenge:	Existing domain adaptation algorithms for text classification are limited by lack of training data and exploiting domain idiosyncrasies to improve performance.
Approach:	They propose a domain adaptation layer that learns weights to combine a generic and a specific word embedding into a DA embeddable.
Outcome:	The proposed approach improves on binary and multi-class classification tasks using popular encoder architectures.

Copied to clipboard

Denis McInerney, Geoffrey Young, Jan-Willem van de Meent, Byron Wallace

On the Influence of Masking Policies in Intermediate Pre-training (2021.emnlp-main)

Challenge:	a recent study suggests that linear models with interpretable features are more reliable than opaque models.
Approach:	They propose an approach for natural-language specification of features for linear models . they prompt LLMs with expert-crafted queries to generate interpretable features from health records .
Outcome:	The proposed approach can be used to craft features clinically meaningful for downstream tasks . it is based on a risk prediction task and standard predictive tasks based upon this data .

Copied to clipboard

Qinyuan Ye, Belinda Z. Li, Sinong Wang, Benjamin Bolte, Hao Ma, Wen-tau Yih, Xiang Ren, Madian Khabsa

VDebugger: Harnessing Execution Feedback for Debugging Visual Programs (2024.findings-emnlp)

Challenge:	Existing studies show that inserting an intermediate pre-training stage improves performance of masked language models.
Approach:	They propose methods to automate the discovery of optimal masking policies via direct supervision or meta-learning.
Outcome:	The proposed method outperforms the heuristic of masking named entities on TriviaQA and can be generalizable beyond that task.

Copied to clipboard

Xueqing Wu, Zongyu Lin, Songyan Zhao, Te-Lin Wu, Pan Lu, Nanyun Peng, Kai-Wei Chang

Model-tuning Via Prompts Makes NLP Models Adversarially Robust (2023.emnlp-main)

Challenge:	Visual programs are executable code generated by large language models to address visual reasoning problems.
Approach:	They propose a critic-refiner framework that localizes and debugs visual programs by tracking execution step by step.
Outcome:	The proposed framework detects and corrects program errors leveraging detailed execution feedback, improving interpretability and accuracy.

Copied to clipboard

Mrigank Raman, Pratyush Maini, J Kolter, Zachary Lipton, Danish Pruthi

Neuro-Symbolic Sentiment Analysis with Dynamic Word Sense Disambiguation (2023.findings-emnlp)

Challenge:	Pre-trained models are typically adapted to downstream tasks by appending a randomly initialized multilayer perceptron to their topmost representation layer and fine-tuning the entire model on a downstream task.
Approach:	They propose to append a multilayer perceptron to a CLS token and fine-tune the entire model on a downstream task.
Outcome:	The proposed model-tuning via prompts outperforms adversarial training-based state-of-art defenses by 3.5% and improves against adversarials by 8% over standard methods.

Copied to clipboard

Xulang Zhang, Rui Mao, Kai He, Erik Cambria

Explainability & Interpretability in NLP

Challenge:	Traditional neural network models represent word senses as vectors that are uninterpretable for humans.
Approach:	They propose a framework that incorporates word Sense Disambiguation (WSD) by identifying and paraphrasing ambiguous words to improve sentiment predictions.
Outcome:	The proposed framework improves sentiment analysis accuracy and interpretability on a downstream task without ground-truth word sense labels.

Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles (2024.emnlp-main)

Copied to clipboard

Ryan Louie, Ananjan Nandi, William Fang, Cheng Chang, Emma Brunskill, Diyi Yang

"I Don’t Know What to Say": A Fact-Filling Questionnaire Method to Help Non-Experts Talk to LegalAI Assistant (2026.findings-acl)

Challenge:	Existing methods for improving LLMs in simulations are limited due to privacy concerns and limited domain knowledge.
Approach:	They propose a pipeline that elicits qualitative feedback from a domain-expert and transforms it into a set of principles that govern an LLM-prompted roleplay.
Outcome:	The proposed pipeline shows a 30% improvement in response quality and principle following for the downstream task.

Copied to clipboard

Yuting Huang, Yiquan Wu, Meitong Guo, Ang Li, Xiaozhong Liu, Keting Yin, Fei Wu, Kun Kuang

Natural Language Interfaces

Challenge:	Recent advances in large language models (LLMs) have greatly expanded the scope of legal AI.
Approach:	They propose a method that generates questionnaires to help users refine queries . they leverage an iterative training process that collects valuable questionnaires .
Outcome:	The proposed method improves the completeness of queries and ensures the performance of domain-specific models in downstream legal tasks.

Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognition (2023.findings-emnlp)

Copied to clipboard

Isaac Slaughter, Craig Greenberg, Reva Schwartz, Aylin Caliskan

Semi-Structured Object Sequence Encoders (2023.findings-emnlp)

Challenge:	Existing work has established that a person’s demographics and speech style affect how well speech processing models perform for them.
Approach:	They propose a method to detect bias in pre-trained models by using word embedding association tests in natural language processing to quantify bias in models' representations of different concepts.
Outcome:	The proposed method detects bias in pre-trained models and can have real-world effects.

Copied to clipboard

Rudra Murthy, Riyaz Bhat, Chulaka Gunasekara, Siva Patel, Hui Wan, Tejas Dhamecha, Danish Contractor, Marina Danilevsky

Unlabeled Debiasing in Downstream Tasks via Class-wise Low Variance Regularization (2024.emnlp-main)

Challenge:	Semi-structured object sequences are often represented as a sequence of key-value pairs over time . authors propose a two-part approach that takes each key independently and encodes a representation of its values over time.
Approach:	They propose a two-part approach that first considers each key independently and encodes a representation of its values over time.
Outcome:	The proposed approach outperforms existing methods on multiple prediction tasks using real-world data.

Copied to clipboard

Shahed Masoudian, Markus Frohmann, Navid Rekabsaz, Markus Schedl

Dynamic Low-rank Estimation for Transformer-based Language Models (2023.findings-emnlp)

Challenge:	Existing methods for debiasing depend on attribute labels and target attributes.
Approach:	They propose a method that uses class-wise variance of embeddings to reduce the effects of debiasing on a downstream task.
Outcome:	The proposed method outperforms baselines that rely on attribute labels while maintaining performance on the target task.

Copied to clipboard

Ting Hua, Xiao Li, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, Hongxia Jin

ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness (2023.emnlp-main)

Challenge:	RankDyna is a matrix decomposition method that can be used to compress Transformer-based language models.
Approach:	They propose a matrix decomposition method that enables dynamic rank resource allocation . they say it can outperform current SOTA methods under various parameter budget levels .
Outcome:	The proposed method outperforms current SOTA methods under various budget levels . the proposed method is more efficient with higher compression rates .

Copied to clipboard

Archiki Prasad, Swarnadeep Saha, Xiang Zhou, Mohit Bansal

Fast Adaptation via Prompted Data: An Efficient Cross-Domain Fine-tuning Method for Large Language Models (2024.lrec-main)

Challenge:	Existing methods focus on whether the reasoning chain leads to the correct conclusion, but this view may confound reasoning quality with other spurious shortcuts to predict the answer.
Approach:	They propose a framework that evaluates reasoning chains via two key properties: (1) correctness, i.e., each step makes a valid inference based on information contained within the step, preceding steps, and input context, and (2) informativeness, respectively.
Outcome:	The proposed framework evaluates reasoning chains via two key properties: (1) correctness, i.e., each step makes a valid inference based on information contained within the step, preceding steps, and input context, and (2) informativeness, which is helpful towards deriving the generated answer.

Copied to clipboard

Yiming Zhang, Hantao Yang, Haobo Wang, Jake Zhao

Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Models (2024.findings-emnlp)

Challenge:	Large language models (LLMs) have been successful in a variety of natural language understanding tasks, but domain discrepancies between the downstream task and the pre-training corpora may have hindered LLMs to excel further in the vertical applications.
Approach:	They propose a Fast Adaptation method for LLMs via Prompted Data that integrates downstream text corpora, gold labels and external knowledge sources into a highly controllable prompt.
Outcome:	The proposed method bridges the gap between the downstream task and the pre-training corpora and integrates downstream text corpors, gold labels and external knowledge sources into a highly controllable prompt.

Copied to clipboard

Xu Han, Linghao Jin, Xuezhe Ma, Xiaofeng Liu

Revisiting Pruning vs Quantization for Small Language Models (2025.findings-emnlp)

Challenge:	Existing fine-tuning algorithms for vision-language models are restricted by patient privacy concerns and can contain imperceptible noise.
Approach:	They propose a framework to mitigate adversarial noise and mitigate upstream noise during fine-tuning.
Outcome:	The proposed framework improves model robustness and transferability while decreasing noise levels negatively impact downstream performance.

Copied to clipboard

Zihan Zhou, Simon Kurz, Zhixue Zhao

Enjoy the Salience: Towards Better Transformer-based Faithful Explanations with Word Salience (2021.emnlp-main)

Challenge:	Compressing Small Language Models (SLMs) is particularly suited for resource-constrained devices, but their compression dynamics remain underexplored compared to Large Language Model (LLMs).
Approach:	They evaluated post-training pruning and quantization methods across six SLMs from 0.5 to 3.8B, seven languages, and seven downstream tasks.
Outcome:	The proposed methods outperform pruning and quantization on six SLMs from 0.5 to 3.8B, seven languages, and seven downstream tasks.

Copied to clipboard

George Chrysostomou, Nikolaos Aletras

Explainability & Interpretability in NLP

Challenge:	Pretrained transformer-based language models have demonstrated state-of-the-art predictive performance when adapted into a range of language understanding tasks.
Approach:	They propose to use salient information extracted a priori from training data to complement the task-specific information learned by the model during fine-tuning on a downstream task.
Outcome:	The proposed model can provide more faithful explanations across four different feature attribution methods compared to vanilla BERT.

Map of Encoders – Mapping Sentence Encoders using Quantum Relative Entropy (2026.acl-long)

Copied to clipboard

Gaifan Zhang, Danushka Bollegala

Enhancing Distractor Generation for Multiple-Choice Questions with Retrieval Augmented Pretraining and Knowledge Graph Integration (2024.findings-acl)

Challenge:	a method to compare and visualise sentence encoders at scale is proposed . we map encoder LLMs using QRE-based feature vectors, which are then projected to 2D .
Approach:	They propose a method to compare and visualise sentence encoders at scale by creating a map of encoder . they construct a QRE-based map of sentences covering 1101 publicly available sentence encoded sentences .
Outcome:	The proposed method compares sentence encoders at scale by creating a map of encoder models . it shows that the map accurately reflects relationships between encoder and unit base encoder .

Copied to clipboard

Han Cheng Yu, Yu An Shih, Kin Man Law, KaiYu Hsieh, Yu Chen Cheng, Hsin Chih Ho, Zih An Lin, Wen-Chuan Hsu, Yao-Chung Fan

A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models (2023.emnlp-main)

Challenge:	Existing LMs undergo task-agnostic pertaining, but task-specific pretraining has gained prominence.
Approach:	They propose retrieval augmented pretraining and task-specific pretraining for DG . they propose to refine language model pretraining to align it more closely with downstream task .
Outcome:	The proposed method improves the performance of multiple-choice questions by integrating knowledge graphs and language models.

Copied to clipboard

Yi Zhou, Jose Camacho-Collados, Danushka Bollegala

LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks (2024.acl-long)

Challenge:	Various types of social biases have been reported with pretrained Masked Language Models (MLMs) in prior work.
Approach:	They conduct a comprehensive study on 39 pretrained MLMs to examine their model factors and their social biases.
Outcome:	The proposed model factors influence social biases learned by an MLM and their downstream task performance.

Copied to clipboard

Hanqing Wang, Bowen Ping, Shuo Wang, Xu Han, Yun Chen, Zhiyuan Liu, Maosong Sun

Investigating the Effect of Pre-finetuning BERT Models on NLI Involving Presuppositions (2023.findings-emnlp)

Challenge:	LoRA-Flow uses lightweight modules to customize large language models for downstream tasks . previous work on LoRA combination relied on task-level weights for each involved LoRA .
Approach:	They propose a LoRA-Flow approach that uses dynamic weights to adjust the impact of different LoRAs.
Outcome:	The proposed method outperforms baselines with task-level weights on six generative tasks.

Copied to clipboard

Jad Kabbara, Jackie Cheung

AMAL: Meta Knowledge-Driven Few-Shot Adapter Learning (2022.emnlp-main)

Challenge:	a study of presupposition, discourse and sarcasm suggests that pre-finetuning can improve models' performance on presimplified cases.
Approach:	They propose to leverage the connection between presupposition, discourse and sarcasm to improve models' performance.
Outcome:	The proposed model improves on cases involving presupposition by pre-finetuning on additional tasks and datasets.

Copied to clipboard

S. K. Hong, Tae Young Jang

Target-Aware Language Modeling via Granular Data Sampling (2024.emnlp-main)

Challenge:	Existing methods for fine-tuning pre-trained language models fail to yield meaningful results in the few-shot regime.
Approach:	They propose a meta-learning-driven low-rank adapter pooling method for leveraging pre-trained language models even with just a few data points.
Outcome:	The proposed method outperforms previous few-shot learning methods on five text classification benchmark datasets.

Copied to clipboard

Ernie Chang, Pin-Jie Lin, Yang Li, Changsheng Zhao, Daeil Kim, Rastislav Rabatin, Zechun Liu, Yangyang Shi, Vikas Chandra

An Empirical Study on the Transferability of Transformer Modules in Parameter-efficient Fine-tuning (2022.emnlp-main)

Challenge:	Language model pretraining is the cornerstone of universal language models (LMs), creating generalpurpose representations to excel across a variety of downstream tasks.
Approach:	They propose to use multi-granular tokens to sample large-scale language models for domain-specific use cases.
Outcome:	The proposed model outperforms random sampled samples on eight benchmarks with 1% of the data and performs on par with the full RefinedWeb data.

Copied to clipboard

Mohammad AkbarTajari, Sara Rajaee, Mohammad Taher Pilehvar

Extrinsic Evaluation of Machine Translation Metrics (2023.acl-long)

Challenge:	Parameter-efficient fine-tuning is a computationally expensive process . introducing new parameters to an already-large model can be considered a drawback.
Approach:	They investigate the capability of different transformer modules in transferring knowledge from a pre-trained model to a downstream task.
Outcome:	The proposed methods show that each transformer module is a winning ticket . they show that with only 0.003% updateable parameters, they can show acceptable performance on target tasks.

Copied to clipboard

Nikita Moghe, Tom Sherborne, Mark Steedman, Alexandra Birch

Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking (2023.findings-acl)

Challenge:	MT metrics are widely used to distinguish the quality of machine translation systems across relatively large test sets.
Approach:	They evaluate the segment-level performance of the most widely used MT metrics by correlating them with how useful they are for downstream tasks.
Outcome:	The MT metrics are widely used to distinguish the quality of machine translation systems across relatively large test sets.

Copied to clipboard

Keshav Santhanam, Jon Saad-Falcon, Martin Franz, Omar Khattab, Avi Sil, Radu Florian, Md Arafat Sultan, Salim Roukos, Matei Zaharia, Christopher Potts

LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering (2025.acl-long)

Challenge:	Neural information retrieval (IR) systems have progressed rapidly in recent years . many IR benchmarks focus on downstream task accuracy, concealing costs incurred .
Approach:	They propose to include efficiency considerations on IR benchmarks to help drive progress . eral et al. propose to incorporate query latency and cost budgets into evaluation .
Outcome:	a new study shows that the best IR system varies according to how efficiency considerations are chosen and weighed . the proposed benchmarks would allow for more thorough exploration of possible system designs .

Copied to clipboard

Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, Yunpu Ma

SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models (2024.findings-emnlp)

Challenge:	Multimodal Large Language Models (MLLMs) enhance visual tasks by integrating visual representations into large language models.
Approach:	They propose a method to re-balance modalities by steering visual representations . they propose LLaVA Steering, a platform that enables rapid customization of MLLMs a component-based architecture .
Outcome:	The proposed model re-balances the modalities of visual representations in large language models . the model requires 500 times fewer trainable parameters than LoRA while maintaining comparable performance .

Copied to clipboard

Juan Munoz, Jinjie Yuan, Nilesh Jain

Learning Together to Perform Better: Teaching Small-Scale LLMs to Collaborate via Preferential Rationale Tuning (2025.acl-long)

Challenge:	Large pre-trained models are often adapted to a desired domain or task through a fine-tuning stage.
Approach:	They propose an end-to-end solution for sparse parameter-efficient fine-tuning of large pre-trained models.
Outcome:	The proposed approach can be used to combine sparse weights with low-rank adapters without losing sparsity and accuracy.

Copied to clipboard

Sohan Patnaik, Milan Aggarwal, Sumit Bhatia, Balaji Krishnamurthy

XPrompt: Exploring the Extreme of Prompt Tuning (2022.emnlp-main)

Challenge:	Prior studies have demonstrated that LLMs generate step-by-step rationales, but limited data is available to improve their performance in commercial settings due to copyright and legal issues.
Approach:	They propose a trainable framework that tunes a (small) LLM to generate outputs from a pool of diverse rationales that selectively improves the downstream task.
Outcome:	The proposed framework outperforms several trainable and prompting baselines on maths problem solving, natural language inference, and commonsense reasoning.

Copied to clipboard

Fang Ma, Chen Zhang, Lei Ren, Jingang Wang, Qifan Wang, Wei Wu, Xiaojun Quan, Dawei Song

Image Matters: A New Dataset and Empirical Study for Multimodal Hyperbole Detection (2024.lrec-main)

Challenge:	Prompt tuning learns soft prompts to condition pre-trained Language Models for performing downstream tasks in a parameter-efficient manner.
Approach:	They propose a Prompt tuning model with an eXtremely small scale that learns soft prompts to condition the frozen Pre-trained Language Models for performing downstream tasks in a parameter-efficient manner.
Outcome:	The proposed model outperforms the vanilla Prompt-Tuning and can significantly improve across tasks and model scales.

Copied to clipboard

Huixuan Zhang, Xiaojun Wan

Impact of Task Adapting on Transformer Models for Targeted Sentiment Analysis in Croatian Headlines (2024.lrec-main)

Challenge:	linguistic detection of hyperbole is an important part of understanding human expression . studies on hyperbolic expressions focus on text modality, but social media can be used to detect it .
Approach:	They propose to use a multimodal detection dataset to study hyperbole detection . they treat text and image as two modalities and evaluate pre-trained encoders .
Outcome:	The proposed dataset is constructed from five different keywords and shows its performance.

Copied to clipboard

Sofia Lee, Jelke Bloem

Fair Text-Attributed Graph Representation Learning (2025.findings-emnlp)

Challenge:	BERT models are often taken off-the-shelf and fine-tuned on a downstream task.
Approach:	They propose an extra stage of self-supervised task-adaptive pre-training to perform a task on a number of Croatian-supporting Transformer models.
Outcome:	The proposed approach improves performance across multilingual models but not in Croatian-dominant models.

Copied to clipboard

Ruilin Luo, Tianle Gu, Lin Wang, Yunfeng Zhou, Songtao Jiang, Lei Wang, Yujiu Yang

Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models (2022.lrec-1)

Challenge:	Text-Attributed Graphs (TAGs) inherit issues from Graph Neural Networks such as fairness.
Approach:	They propose to evolve LM-as-encoder to LM as-fair-encoding process to explore fairness in TAGRL.
Outcome:	The proposed process can be integrated with fairness-enhancing strategies on the GNNs decoder side.

Copied to clipboard

Keigo Takahashi, Danushka Bollegala

Rethinking the Evaluation of In-Context Learning for LLMs (2024.emnlp-main)

Challenge:	Existing methods for creating metaembeddings from static word embeddings have been proposed, but they are not tied to a particular downstream task.
Approach:	They propose a sentence-level meta-embedding learning method that takes contextualised word embedding models and learns a phrase embeddable that preserves complementary strengths of the input source NLMs.
Outcome:	The proposed method outperforms existing methods on semantic textual similarity benchmarks on a supervised baseline and on token-level embeddings.

Copied to clipboard

Guoxin Yu, Lemao Liu, Mo Yu, Yue Yu, Xiang Ao

Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation (2023.acl-long)

Challenge:	Existing studies evaluate In-context learning methods based on task performance . however, this evaluation protocol overlooks the significant cost associated with the demonstration configuration process .
Approach:	They propose a two-dimensional evaluation paradigm that considers both configuration costs and task performance.
Outcome:	The proposed evaluation paradigm can be applied to any ICL method as a plugin.

Copied to clipboard

Chaoya Jiang, Wei Ye, Haiyang Xu, Songfang Huang, Fei Huang, Shikun Zhang

Improving Low-Resource Languages in Pre-Trained Multilingual Language Models (2022.emnlp-main)

Challenge:	Large-scale pre-trained vision-language models have recently achieved tremendous success on a wide range of cross-modal tasks.
Approach:	They propose a new framework for a semantically-aware contrastive learning that minimizes the MI between false negative and positive samples .
Outcome:	The proposed framework minimizes the MI between false negative samples and positive samples even though they share similar semantics.

Copied to clipboard

Viktor Hangya, Hossain Shaikh Saadi, Alexander Fraser

LLMs cannot find reasoning errors, but can correct them given the error location (2024.findings-acl)

Challenge:	Pre-trained multilingual language models are the foundation of many NLP approaches, but are often not well-supported by these models due to small available monolingual corpora.
Approach:	They propose an unsupervised approach to improve cross-lingual representations of low-resource languages by bootstrapping word translation pairs from monolingual corpora and using them to improve language alignment.
Outcome:	The proposed approach improves cross-lingual representations on low-resource languages using word retrieval and zero-shot named entity recognition.

Copied to clipboard

Gladys Tyen, Hassan Mansoor, Victor Carbune, Peter Chen, Tony Mak

The Price of Format: Diversity Collapse in LLMs (2025.findings-emnlp)

Challenge:	Recent attempts to self-correct logical or reasoning errors often cause correct answers to become incorrect, resulting in poor performance overall.
Approach:	They propose to use a backtracking setup to test the correction abilities of LLMs on their mistake-finding ability to find logical mistakes.
Outcome:	The proposed model improves on 5 reasoning tasks, showing that it can correct logical mistakes without ground truth labels or training data.

Copied to clipboard

Longfei Yun, Chenyang An, Zilong Wang, Letian Peng, Jingbo Shang

From Human Reading to NLM Understanding: Evaluating the Role of Eye-Tracking Data in Encoder-Based Models (2025.acl-long)

Challenge:	Instruction-tuned large language models employ structured templates to enforce format consistency during inference.
Approach:	They fine-tune instruction-tuning large language models with structured templates and evaluate their results across three axes: downstream task performance, alignment behavior, and output diversity.
Outcome:	The proposed model generates semantically similar outputs even under high temperature sampling and structural tokens in templates significantly constrain the model’s output space.

Copied to clipboard

Luca Dini, Lucia Domenichelli, Dominique Brunato, Felice Dell’Orletta

Difference-Masking: Choosing What to Mask in Continued Pretraining (2023.findings-emnlp)

Challenge:	integrating eye-tracking features into Neural Language Models does not degrade downstream task performance, enhances alignment between model attention and human attention patterns, and compresses the embedding space.
Approach:	They used eye-gaze data from the Ghent Eye-Tracking Corpus to investigate how integrating knowledge of human reading behavior impacts Neural Language Models.
Outcome:	The proposed approach does not degrade downstream task performance, enhances alignment between model attention and human attention patterns, and compresses the embedding space.

Copied to clipboard

Alex Wilf, Syeda Akter, Leena Mathur, Paul Liang, Sheryl Mathew, Mengrou Shou, Eric Nyberg, Louis-Philippe Morency

Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression (2024.findings-emnlp)

Challenge:	Existing approaches to masked prediction have shown that deciding what to mask can substantially improve learning outcomes.
Approach:	They propose a masking strategy that automatically chooses what to mask during continued pretraining by considering what makes a task domain different from the pretraining domain.
Outcome:	The proposed masking strategy outperforms baselines on language-only and multimodal video tasks.

Copied to clipboard

Zhichao Xu, Ashim Gupta, Tao Li, Oliver Bentham, Vivek Srikumar

Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Merging (2025.findings-emnlp)

Challenge:	Prior work on compression prioritizes preserving perplexity, which is analogous to training loss.
Approach:	They examine the impact of model compression along four dimensions: degeneration harm, representational harm, dialect bias, and language modeling and downstream task performance.
Outcome:	The proposed compression methods can lead to unexpected consequences, the authors show . quantization preserves bias while pruning degrades quickly.

Copied to clipboard

Hua Farn, Hsuan Su, Shachi H. Kumar, Saurav Sahay, Shang-Tse Chen, Hung-yi Lee

LoNAS: Elastic Low-Rank Adapters for Efficient Large Language Models (2024.lrec-main)

Challenge:	Fine-tuning large language models for downstream tasks often leads to catastrophic forgetting, notably degrading the safety of original alignments.
Approach:	They propose to merge the weights of pre- and post-fine-tuned models to improve safety while enhancing performance.
Outcome:	Experiments across different downstream tasks and models validate the method’s practicality and effectiveness.

Copied to clipboard

Juan Pablo Munoz, Jinjie Yuan, Yi Zheng, Nilesh Jain

Group-SAE: Efficient Training of Sparse Autoencoders for Large Language Models via Layer Groups (2025.emnlp-main)

Challenge:	Large Language Models (LLMs) reach hundreds of billions of parameters and require resources for training and inference stages.
Approach:	They propose a low-rank adapter to reduce the number of trainable parameters in a model and reduce memory requirements.
Outcome:	The proposed approach reduces memory and compute requirements while preserving performance.

Copied to clipboard

Davide Ghilardi, Federico Belotti, Marco Molinari, Tao Ma, Matteo Palmonari

Task-Aware Self-Supervised Framework for Dialogue Discourse Parsing (2023.findings-emnlp)

Challenge:	Sparse Autoencoders (SAEs) are a promising unsupervised approach for understanding the representations of layers of Large Language Models (LLMs).
Approach:	They propose a method that groups similar models and trains a single SAE per group based on representational similarity across layers.
Outcome:	Experiments on Pythia family models show that the proposed method significantly accelerates training with minimal impact on reconstruction quality and comparable downstream task performance and interpretability over baseline SAEs trained layer by layer.

Copied to clipboard

Wei Li, Luyao Zhu, Wei Shao, Zonglin Yang, Erik Cambria

Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond (2023.emnlp-main)

Challenge:	Existing discourse parsing approaches are constrained by predefined relation types, which can impede the adaptability of the parser for downstream tasks.
Approach:	They propose to introduce a task-aware paradigm to improve the versatility of the parser.
Outcome:	Empirical studies on dialogue discourse parsing datasets and a downstream task demonstrate the proposed framework.

Copied to clipboard

Siyang Liu, Naihao Deng, Sahand Sabour, Yilin Jia, Minlie Huang, Rada Mihalcea

Understanding and Improving Information Preservation in Prompt Compression for LLMs (2025.findings-emnlp)

Challenge:	Existing methods to generate text in mental health are limiting, but they are effective for many tasks.
Approach:	They propose a task-adaptive tokenizer that allows for the integration of task-specific tokens into the pre-trained model's tokenization step.
Outcome:	The proposed tokenization approach improves generation performance on psychological question-answering tasks in Chinese and English while using 60% fewer tokens.

Copied to clipboard

Weronika Łajewska, Momchil Hardalov, Laura Aina, Neha Anna John, Hang Su, Lluis Marquez

Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning (2024.emnlp-main)

Challenge:	Recent advances in large language models have enabled their successful application to a broad range of tasks.
Approach:	They propose a framework that allows for in-depth analysis of prompt compression methods.
Outcome:	The proposed framework analyzes state-of-the-art soft and hard compression methods . it shows that some fail to preserve key details from the original prompt, limiting performance on complex tasks.

Copied to clipboard

Abhinav Bandari, Lu Yin, Cheng-Yu Hsieh, Ajay Jaiswal, Tianlong Chen, Li Shen, Ranjay Krishna, Shiwei Liu

Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition (2024.lrec-main)

Challenge:	Existing approaches to prune LLMs rely on the C4 dataset as calibration data . arithmetic datasets perform better than pre-training datasets for pruning, whereas chain-of-thought is only useful on certain tasks.
Approach:	They evaluate the selection of calibration data for LLM pruning across a wide range of datasets . they find that C4 is not the optimal calibration data, and that CoT is only useful on certain tasks.
Outcome:	The chosen calibration data significantly impacts the performance of pruned LLMs, the authors found . their results shed light on the importance of carefully selecting calibration data for LLM pruning .

Copied to clipboard

Yash Jain, David M. Chan, Pranav Dheram, Aparna Khare, Olabanji Shonibare, Venkatesh Ravichandran, Shalini Ghosh

Towards Informative Few-Shot Prompt with Maximum Information Gain for In-Context Learning (2023.findings-emnlp)

Challenge:	Existing methods for pre-training for automatic speech recognition (ASR) focus on single-stage pre-train followed by fine-tuning on downstream task.
Approach:	They propose a multi-modal pre-training method that combines unsupervised pre-training with translation-based supervised mid-training.
Outcome:	The proposed method improves WERs by 38.45% over baselines on both Librispeech and SUPERB.

Copied to clipboard

Hongfu Liu, Ye Wang

New Evaluation Methodology for Qualitatively Comparing Classification Models (2024.lrec-main)

Challenge:	Large Language models (LLMs) have the capability to engage In-context Learning (ICL) however, this particular learning paradigm suffers from high instability stemming from factors such as input distribution, order and prompt formats.
Approach:	They propose to quantify the information gain obtained in prediction after observing a given example candidate and to sample those with maximum IG.
Outcome:	The proposed method can yield an average relative improvement of 14.3% across six classification tasks using three LLMs.

Copied to clipboard

Ahmad Aljanaideh

NLoPT: N-gram Enhanced Low-Rank Task Adaptive Pre-training for Efficient Language Model Adaption (2024.lrec-main)

Challenge:	Text Classification is one of the most common tasks in Natural Language Processing.
Approach:	They propose a method for performing qualitative assessment over multiple classification models using a fine-tuned BERT and Logistic Regression evaluation methodology.
Outcome:	The proposed evaluation methodology outperforms the baseline model in linguistic clustering and Sentiment Analysis.

Copied to clipboard

Hao Gu, Jiangyan Yi, Zheng Lian, Jianhua Tao, Xinrui Yan

IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation (2026.acl-long)

Challenge:	Pre-trained Language Models (PLMs) have superior performance on downstream tasks . however, conventional TAPT adjusts all parameters of the PLMs, which distorts the learned generic knowledge embedded in the original PLM's weights.
Approach:	They propose a two-step n-gram enhanced low-rank task adaptive pre-training method to customize a PLM to the downstream task.
Outcome:	The proposed method improves performance on six datasets from four domains.

Copied to clipboard

Bosi Wen, Yilin Niu, Cunxiang Wang, Xiaoying Ling, Ying Zhang, Pei Ke, Hongning Wang, Minlie Huang

Challenge:	Existing benchmarks for instruction-following lack data coverage and oversimplified pairwise evaluation paradigms that misalign with model optimization scenarios.
Approach:	They propose a meta-evaluation benchmark for instruction-following that covers diverse instruction and constraint types and a preference graph for each instruction.
Outcome:	Extensive experiments on IF-RewardBench show that the proposed benchmark achieves a stronger positive correlation with downstream task performance compared to existing benchmarks.

Advancing Collaborative Debates with Role Differentiation through Multi-Agent Reinforcement Learning (2025.acl-long)

Copied to clipboard

Haoran Li, Ziyi Su, Yun Xue, Zhiliang Tian, Yiping Song, Minlie Huang

Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning (2025.emnlp-main)

Challenge:	Multi-agent collaborative tasks exhibit exceptional capabilities in natural language applications and generation.
Approach:	They propose a multi-LLM Cooperation framework with automatic role assignment capabilities that allows multiple agents to embed roles in turn-based speaking.
Outcome:	The proposed framework improves collaboration and expertise among agents and teams by enabling them to share roles and develop complementary strengths from the optimization level.

Copied to clipboard

Wei Huang, Anda Cheng, Yinggui Wang

Pruning Weights but Not Truth: Safeguarding Truthfulness While Pruning LLMs (2025.findings-emnlp)

Challenge:	Recent advances in large language models (LLMs) have shown impressive capabilities in various downstream tasks but typically face Catastrophic Forgetting (CF) during fine-tuning.
Approach:	They propose a pruning-based approach to balance CF and downstream task performance by integrating the ratio of the task vector to pre-trained model parameters into the pruning criteria.
Outcome:	The proposed pruning-based approach limits CF to just 0.25% while maintaining 99.67% accuracy on downstream tasks.

Copied to clipboard

Yao Fu, Runchao Li, Xianxuan Long, Haotian Yu, Xiaotian Han, Yu Yin, Pan Li

Challenge:	Neural network pruning disrupts LLMs’ internal activation features crucial for lie detection . layer-wise pruning sparsity inadvertently removes crucial weights, failing to improve lie detection performance despite its reliance on the most crucial LLM layer.
Approach:	They propose a pruning approach that places greater emphasis on layers with more activation outliers and stronger discriminative features simultaneously.
Outcome:	The proposed approach improves the hallucination detection for pruned LLMs (achieving 88% accuracy at 50% sparsity) and enhances their performance on TruthfulQA.

TELL-TALE: Task Efficient LLMs with Task Aware Layer Elimination (2026.findings-acl)

Copied to clipboard

Omar Naim, Krish Sharma, Niyar R Barman, Nicholas Asher

Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law (2025.acl-long)

Challenge:	Large Language Models typically come with a fixed architecture, but not all layers contribute equally to every downstream task.
Approach:	They propose an inference-time method that selectively removes irrelevant or detrimental layers . the method is hardware-agnostic, requires no retraining, and operates entirely at inference time .
Outcome:	The proposed method matches or surpasses baseline performance while reducing computational costs.

Copied to clipboard

Qiming Ge, Shuhao Xing, Songyang Gao, Yunhua Zhou, Yicheng Zou, Songyang Zhang, Zhi Chen, Hang Yan, Qi Zhang, Qipeng Guo, Kai Chen

GottBERT: a pure German Language Model (2024.emnlp-main)

Challenge:	Large language models have demonstrated impressive performance across a wide range of tasks, but this achievement comes with the trade-off of significant computational demands.
Approach:	They propose a scaling law that decomposes the overall validation loss and assigns different importance weights to tokens to assess a specific meta-capability.
Outcome:	The proposed model can predict the loss trending of models across different levels of computation without a gap between validation loss and model's downstream capabilities.

Copied to clipboard

Raphael Scheible, Johann Frei, Fabian Thomczyk, Henry He, Patric Tippmann, Jochen Knaus, Victor Jaravine, Frank Kramer, Martin Boeker

Instability in Downstream Task Performance During LLM Pretraining (2025.findings-emnlp)

Challenge:	Pre-trained language models have advanced natural language processing (NLP) despite the introduction of BERT, single-language models are still relevant.
Approach:	They present a German singlelanguage RoBERT model pre-trained exclusively on the German portion of the OSCAR dataset.
Outcome:	The GottBERT model outperforms the existing models on Named Entity Recognition and text classification tasks.

Copied to clipboard

Yuto Nishida, Masaru Isonuma, Yusuke Oda

Causal Graph based Event Reasoning using Semantic Relation Experts (2025.acl-long)

Challenge:	a study of large language models shows that task scores fluctuate throughout training .
Approach:	They empirically analyze the stability of downstream task performance in an LLM .
Outcome:	The proposed methods improve performance stability without changes to the training procedure.

Copied to clipboard

Mahnaz Koupaee, Xueying Bai, Mudan Chen, Greg Durrett, Nathanael Chambers, Niranjan Balasubramanian

OASIS: Mitigating Harmful Fine-tuning Attacks on LLMs via Orthogonal and Adaptive Safety Alignment Strategy (2026.acl-long)

Challenge:	Recent advances in event reasoning have limited ability to accurately identify causal connections between events.
Approach:	They propose a collaborative approach to generate correct graphs and graphs to assist reasoning . they propose 'a causal chain of events' task that requires a causal link between events .
Outcome:	The proposed approach achieves competitive results with state-of-the-art models on forecasting and next event prediction tasks.

Copied to clipboard

Jiayu Tang, Guowei Peng, Qiuhao Xie, Yuning Yang, Xiurui Xie, Guisong Liu

RShield: A User-level Traceable Backdoor Watermark for LLMs in Embedding-as-a-Service (2026.findings-acl)

Challenge:	Existing methods to decouple safety enforcement from harmful feature acquisition rely on perturbation directions that conflict with harmful gradients . harmful fine-tuning attacks pose a significant challenge for service providers aiming to uphold rigorous safety standards.
Approach:	They propose an orthogonal and ad hoc safety alignment strategy to decouple safety enforcement from harmful feature acquisition.
Outcome:	Experiments on four large language models show that OASIS reduces the Harmful Score by 60% compared to baselines while maintaining stable task utility.

Copied to clipboard

Lingyun Xiang, Yufan Zhong, Chengfu Ou, Zhihua Xia, Chunfang Yang, Daojian Zeng, Zhangjie Fu

Understanding and Mitigating Bias Inheritance in LLM-based Data Augmentation on Downstream Tasks (2026.acl-long)

Challenge:	Existing backdoor watermarking techniques are limited to zero-bit detection . RShield enables reliable user-level attribution of large language models under model extraction attacks.
Approach:	They propose a multi-bit backdoor watermarking technique that enables reliable user-level attribution of large language models under model extraction attacks.
Outcome:	RShield achieves 100% multi-bit watermark recovery and high semantic fidelity under model extraction attacks compared to existing methods.

Copied to clipboard

Miaomiao Li, Hao Chen, Yang Wang, Tingyuan Zhu, Weijia Zhang, Kaijie Zhu, Kam-Fai Wong, Jindong Wang

Pre-trained Models Perform the Best When Token Distributions Follow Zipf’s Law (2025.emnlp-main)

Challenge:	Generating synthetic datasets via large language models (LLMs) has emerged as promising approach to improve LLM performance.
Approach:	They propose three mitigation strategies to mitigate bias inheritance in LLMs by analyzing real and LLM-augmented data.
Outcome:	The proposed methods can work differently on different tasks and biases.

Copied to clipboard

Yanjin He, Qingkai Zeng, Meng Jiang

DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents (2026.findings-acl)

Challenge:	Existing large language models typically fix a vocabulary size in advance, then use Byte Pair Encoding (BPE) to construct the tokenizer.
Approach:	They propose a method for determining the vocabulary size by analyzing token frequency distributions through Zipf’s law and propose to use it to optimize model performance.
Outcome:	The proposed method improves model efficiency and effectiveness across NLP, genomics, and chemistry.

Copied to clipboard

Kai Shi, Jun Yang, Ni Yang, Binqiang Pan, Qingsong Xie, null Zhangchao, Zhenyu Yang, Tianhuang Su, Haonan Lu

GiLT: Augmenting Transformer Language Models with Dependency Graphs (2026.acl-long)

Challenge:	Mobile Phone Agents (MPAs) have attracted huge attention due to their practicability in a multitude of scenarios.
Approach:	They propose a data mixture optimization solution that extrapolates optimal data mixtures from a trainable network.
Outcome:	The proposed model outperforms existing methods on open-source benchmarks and on open source benchmarks.

Copied to clipboard

Tianyu Huang, Yida Zhao, Chuyan Zhou, Kewei Tu

Correct-Detect: Balancing Performance and Ambiguity Through the Lens of Coreference Resolution in LLMs (2025.emnlp-main)

Challenge:	Recent work focuses on syntactic tree structures of languages, in particular constituency tree structures.
Approach:	They propose a Graph-Infused Layers Transformer Language Model which leverages dependency graphs to augment Transformer language models.
Outcome:	The proposed model achieves better syntactic generalization while maintaining competitive perplexity compared with baseline models.

Copied to clipboard

Amber Shore, Russell Scheinberg, Ameeta Agrawal, So Young Lee

La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America (2025.acl-long)

Challenge:	Large Language Models (LLMs) are intended to reflect human linguistic competencies . but when context is absent or insufficient, ambiguity resolution becomes more tenuous .
Approach:	They propose a CORRECT-DETECT trade-off between large language models and ambiguity detection . they show that large language model models can achieve good performance with minimal prompting .
Outcome:	The proposed models can achieve good performance with minimal prompting in coreference disambiguation and detection of ambiguity in corefertility tasks, but they cannot do both at the same time.

Copied to clipboard

María Grandury, Javier Aula-Blasco, Júlia Falcão, Clémentine Fourrier, Miguel González Saiz, Gonzalo Martínez, Gonzalo Santamaria Gomez, Rodrigo Agerri, Nuria Aldama García, Luis Chiruzzo, Javier Conde, Helena Gomez Adorno, Marta Guerrero Nieto, Guido Ivetta, Natàlia López Fuertes, Flor Miriam Plaza-del-Arco, María-Teresa Martín-Valdivia, Helena Montoro Zamorano, Carmen Muñoz Sanz, Pedro Reviriego, Leire Rosado Plaza, Alejandro Vaca Serrano, Estrella Vallecillo-Rodríguez, Jorge Vallego, Irune Zubiaga

Dynamic Tool Dependency Retrieval for Lightweight Function Calling (2026.findings-acl)

Challenge:	La Leaderboard is the first open-source leaderboard to evaluate generative Large Language Models (LLMs) in languages and language varieties of Spain and Latin America.
Approach:	They propose to use La Leaderboard to evaluate generative Large Language Models in Spanish and Latin America.
Outcome:	La Leaderboard is the first open-source leaderboard to evaluate generative LLMs in languages and language varieties of Spain and Latin America.

Copied to clipboard

Bhrij Patel, Davide Belli, Amir Jalalirad, Maximilian Arnold, Aleksandr Ermolov, Bence Major

BEFT: Bias-Efficient Fine-Tuning of Language Models in Low-Data Regimes (2026.acl-long)

Challenge:	Existing retrieval methods rely on static inputs, failing to capture multi-step tool dependencies and evolving task context.
Approach:	They propose a lightweight retrieval method that conditions on initial query and evolving task context.
Outcome:	The proposed method improves function calling success rates between 23% and 104% compared to state-of-the-art retrieval methods.

Copied to clipboard

Baichuan Huang, Ananth Balashankar, Amir Aminifar

Uncertainty-Aware Contrastive Sentence Embedding With Local Context Representation for Text Classification (2026.findings-acl)

Challenge:	Fine-tuning bias terms of large language models (LLMs) for downstream tasks has gained a lot of attention over the past few years.
Approach:	They extensively evaluate bq, bk, v across a wide range of LLMs . they find that bv generally leads to higher downstream performance in low-data regimes compared to bQ and bK .
Outcome:	The proposed method improves performance across a wide range of LLMs spanning encoder-only and decoder-free architectures up to 6.7B parameters.

Copied to clipboard

Han Liu, Jiaqing Zhan, Zhichao Chen, Qin Zhang

Measuring User’s Mental Models of Speech Translation in Human-AI Collaboration (2026.acl-long)

Challenge:	Existing models for text classification are based on encoder-only transformers and generative pre-trained transformers.
Approach:	They propose an uncertainty-aware contrastive sentence embedding approach that addresses language ambiguity and inter-class separability for a text classification task.
Outcome:	The proposed approach improves classification accuracy on public datasets compared with state-of-the-art methods.

Copied to clipboard

HyoJung Han, Nishant Balepur, Jordan Lee Boyd-Graber, Marine Carpuat

TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models (2026.acl-long)

Challenge:	Existing research on machine translation tools has not revealed how users perceive MT errors and how they evolve through interaction.
Approach:	They propose a framework where users accept MT output or request professional re-translation to answer questions based on information presented in a foreign language.
Outcome:	The proposed framework can predict where the system is likely to be wrong and how it evolves through interaction.

Copied to clipboard

Jinho Choo, JunSeung Lee, Jimyeong Kim, Yeeho Song, S. K. Hong, Yeong-Dae Kwon

Benchmarking Fine-Grained Error Detection in Multimodal Reasoning (2026.acl-long)

Challenge:	Prior mitigation approaches that optimize entire responses operate at the level of entire responses and can lead to unintended degradation of general model capabilities.
Approach:	They propose a fine-tuning framework to mitigate erroneous outputs by localizing and updating the policy at a granular level.
Outcome:	The proposed framework outperforms baselines on multiple multilingual LLMs across diverse languages while preserving task accuracy.

Copied to clipboard

Chi-Min Chan, Han Zhu, Chunyang Jiang, Jiaming Ji, Juntao Dai, Wei Xue, Sirui Han, Yike Guo

Apertus: Democratizing Open and Compliant LLMs for Global Language Environments (2026.acl-long)

Challenge:	Multimodal Process Reward Models (MPRMs) have emerged as a pivotal framework for enhancing the reasoning capabilities of Multimodal Large Language Models.
Approach:	They propose a benchmark specifically designed to evaluate MPRMs’ proficiency in detecting erroneous reasoning steps across diverse error categories.
Outcome:	The proposed model achieves up to 4.8% performance improvement through test-time scaling.

Copied to clipboard

Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pásztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Ďurech, Ido Hakimi, Juan Garcia Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabolčec, Yixuan Xu, Michael Aerni, Badr AlKhamissi, Inés Altemir Marinas, Mohammad Hossein Amani, Matin Ansaripour, Ilia Badanin, Harold Benoit, Emanuela Boros, Nicholas John Browning, Fabian Bösch, Maximilian Böther, Niklas Canova, Camille Challier, Clément Charmillot, Jonathan Coles, Jan Milan Deriu, Arnout Devos, Lukas Drescher, Daniil Dzenhaliou, Maud Ehrmann, Dongyang Fan, Simin Fan, Silin Gao, Miguel Gila, María Grandury, Diba Hashemi, Alexander Miserlis Hoyle, Jiaming Jiang, Mark Klein, Andrei Kucharavy, Anastasiia Kucherenko, Frederike Lübeck, Roman Machacek, Theofilos Ioannis Manitaras, Andreas Marfurt, Kyle Matoba, Simon Matrenok, Henrique Mendonça, Fawzi Roberto Mohamed, Syrielle Montariol, Luca Mouchel, Sven Najem-Meyer, Jingwei Ni, Gennaro Oliva, Matteo Pagliardini, Elia Palme, Andrei Panferov, Léo Paoletti, Marco Passerini, Ivan Pavlov, Auguste Poiroux, Kaustubh Ponkshe, Nathan Ranchin, Javier Rando, Mathieu Sauser, Jakhongir Saydaliev, Mukhammadali Sayfiddinov, Marian Schneider, Stefano Schuppli, Marco Scialanga, Andrei Semenov, Kumar Shridhar, Raghav Singhal, Anna Sotnikova, Alexander Sternfeld, Ayush Kumar Tarun, Paul Teiletche, Jannis Vamvas, Xiaozhe Yao, Hao Zhao, Alexander Ilic, Ana Klimovic, Andreas Krause, Caglar Gulcehre, David Rosenthal, Elliott Ash, Florian Tramèr, Joost VandeVondele, Livio Veraldi, Martin Rajman, Thomas C. Schulthess, Torsten Hoefler, Antoine Bosselut, Martin Jaggi, Imanol Schlag

Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings (2026.acl-long)

Challenge:	Apertus is a fully open suite of large language models (LLMs) designed to address responsibility shortcomings in today’s open model ecosystem, namely data responsibility and global representation.
Approach:	They propose to release a fully open suite of large language models (LLMs) that address data responsibility and global representation shortcomings in today’s open model ecosystem.
Outcome:	The proposed model is pretrained on openly available data and suppresses verbatim recall of data while retaining task performance.

Copied to clipboard

Tomomasa Hara, Hiroto Kurita, Masaaki Imaizumi, Kentaro Inui, Sho Yokoi